Skip to content

ipgallery / devops

Go to a project
Commit d5896d1b by Amir Aharon
Options

flink and edge

parent 0e84d0a8
Inline Side-by-side
Showing with 257 additions and 12 deletions
common/docs/knowledgeBase/pulsar-flink-admin.txt 0 → 100644
PULSAR - FLINK ADMIN:
PULSAR:
This is a pub/sub msg queue (bus) where domains and stream processing jobs communicates via topics
Topic structure (url) is:
[durability://][tenant]/[namespace]/[topic]
e.g: persistent://mcx/activities/activity
Domains topics:
- all our topics are under 'mcx' tenant
- every domain has it's own namespace according to it's service name (default behavior)
e.g activities domain namespace is 'activities'
- the topics are the api's e.g /activity api will have the topic url: 'persistent://mcx/activities/activity'
- every domain creates it's own topic on startup, except activities that also creates the 'public' topics
Streaming jobs:
- all the streaming jobs have topics under 'public' namespace in the 'mcx' tenant
e.g. the cep-activities job subscribe to the topics: 'persistent://mcx/public/activities' and 'persistent://mcx/public/activity'
Docker:
- in the compose file under pulsar service
- there are 3 ports to open/map:
- 80 for web interface ( only for pulsar-standalone or pulsar-dashboard)
- 8080 for admin
- 6650 the actual pubsub port
Web Interface:
- There is an internal web interface if you run pulsar-standalone docker, but it is not
showing the actual topics in real-time
- pulsar-express:
https://github.com/bbonnin/pulsar-express
This is a remote interface that connects to the pulsar (like the db tools for mongo/redis etc')
you can install it using:
npm install pulsar-express -g (add sudo if you're not on root)
when creating connections use the admin port mapping
FLINK:
This is a stream processing framework, is receives data from 'sources' (subscribes to topics),
receive the msgs and process them in real time, and outputs (if there is output) to 'sinks' ( like http or public to topics)
Every processing unit is called a JOB, each job has several tasks that runs on cluster of nodes.
- the flink framework has a job manager and a task manager that has one or more nodes
- the jobs are loaded to the flink from web interface/commandline or from pre-loaded directory
- our jobs are published to the archiva so that they can be downloaded anywhere and uploaded to
the flink framework, the path is http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/[job-name]/[version]/[job-artifact].jar
e.g: http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/activities-cep-job/1.0.0/activities-cep-job-1.0.0-all.jar
- the jobs doesn't create topics on pulsar, so the activities service must be up before uploading the job, and the pulsar
service must be up before starting the activities service
Docker:
- there are to docker services, one for job manager and one for task manager
- port mapping 8081 for web interface
- if the flink is running on a different network/machine then we need to define the domains needed by the jobs
in the extra_hosts
- since the jobs runs under the flink instances, the jobs logs is sent to the consule log which will appear in the instances docker log.
Web Interface:
- in port 8081, under 'Submit new Job' upload the job, click on it and click 'Submit'
- it may fail on first try due to timeouts, try again before searching for errors in the logs
docker/composers/mcx-cep.yml
......@@ -11,7 +11,8 @@ services:
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
extra_hosts:
- "pulsar:172.16.1.244"
- "alerts:172.16.1.244"
- "scp:172.16.1.72"
# volumes:
# - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
networks:
......@@ -31,22 +32,23 @@ services:
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
extra_hosts:
- "pulsar:172.16.1.244"
- "alerts:172.16.1.244"
- "scp:172.16.1.72"
# volumes:
# - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
networks:
- backend
# pulsar:
# image: apachepulsar/pulsar-standalone:2.4.2
# ports:
# - 8080:8080
# - 8082:80
# - 6650:6650
# # volumes:
# # - "/ext/pulsar/data:/pulsar/data"
# networks:
# - backend
pulsar:
image: apachepulsar/pulsar-standalone:2.4.2
ports:
- 8080:8080
- 8082:80
- 6650:6650
# volumes:
# - "/ext/pulsar/data:/pulsar/data"
networks:
- backend
networks:
backend:
driver: bridge
products/MCX/design/Edge/DataFramwork.graphml 0 → 100644
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:java="http://www.yworks.com/xml/yfiles-common/1.0/java" xmlns:sys="http://www.yworks.com/xml/yfiles-common/markup/primitives/2.0" xmlns:x="http://www.yworks.com/xml/yfiles-common/markup/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:y="http://www.yworks.com/xml/graphml" xmlns:yed="http://www.yworks.com/xml/yed/3" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://www.yworks.com/xml/schema/graphml/1.1/ygraphml.xsd">
<!--Created by yEd 3.19.1.1-->
<key attr.name="Description" attr.type="string" for="graph" id="d0"/>
<key for="port" id="d1" yfiles.type="portgraphics"/>
<key for="port" id="d2" yfiles.type="portgeometry"/>
<key for="port" id="d3" yfiles.type="portuserdata"/>
<key attr.name="url" attr.type="string" for="node" id="d4"/>
<key attr.name="description" attr.type="string" for="node" id="d5"/>
<key for="node" id="d6" yfiles.type="nodegraphics"/>
<key for="graphml" id="d7" yfiles.type="resources"/>
<key attr.name="url" attr.type="string" for="edge" id="d8"/>
<key attr.name="description" attr.type="string" for="edge" id="d9"/>
<key for="edge" id="d10" yfiles.type="edgegraphics"/>
<graph edgedefault="directed" id="G">
<data key="d0"/>
<node id="n0">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="67.0" width="176.0" x="243.30859375" y="412.77599334716797"/>
<y:Fill color="#4CAF50" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="17.96875" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="105.8125" x="35.09375" xml:space="preserve" y="24.515625">Data Normalyzer<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
</node>
<node id="n1">
<data key="d5"/>
<data key="d6">
<y:GenericNode configuration="com.yworks.flowchart.dataBase">
<y:Geometry height="108.0" width="176.0" x="477.0" y="271.0"/>
<y:Fill color="#2196F3" color2="#2196F3" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="17.96875" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="36.419921875" x="69.7900390625" xml:space="preserve" y="45.015625">TSDB<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
</y:GenericNode>
</data>
</node>
<node id="n2">
<data key="d4" xml:space="preserve"/>
<data key="d5"/>
<data key="d6">
<y:UMLNoteNode>
<y:Geometry height="84.0" width="155.0" x="77.0" y="223.0"/>
<y:Fill color="#A5D6A7" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="left" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="144.015625" x="5.4921875" xml:space="preserve" y="19.046875">Templates
converting Sensor data
to MCX Entity Data<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
</y:UMLNoteNode>
</data>
</node>
<node id="n3">
<data key="d5"/>
<data key="d6">
<y:GenericNode configuration="com.yworks.flowchart.annotation">
<y:Geometry height="84.0" width="375.0" x="377.5" y="131.22400665283203"/>
<y:Fill color="#81D4FA" color2="#81D4FA" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="left" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="45.90625" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="320.119140625" x="27.4404296875" xml:space="preserve" y="19.046875">Record structure:
measurement,tag-key,tag-value,field-key,field-value
tags, time, geohash,lat,lon, temp,aqi ...<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:StyleProperties>
<y:Property class="java.lang.Byte" name="com.yworks.flowchart.style.orientation" value="0"/>
</y:StyleProperties>
</y:GenericNode>
</data>
</node>
<node id="n4">
<data key="d5"/>
<data key="d6">
<y:GenericNode configuration="com.yworks.flowchart.dataBase">
<y:Geometry height="108.0" width="176.0" x="477.0" y="392.27599334716797"/>
<y:Fill color="#2196F3" color2="#2196F3" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="31.9375" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="97.333984375" x="39.3330078125" xml:space="preserve" y="38.03125">Entity &amp; Spatial
DB<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
</y:GenericNode>
</data>
</node>
<node id="n5">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="67.0" width="176.0" x="710.69140625" y="412.77599334716797"/>
<y:Fill color="#4CAF50" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="17.96875" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="44.283203125" x="65.8583984375" xml:space="preserve" y="24.515625">Query <y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
</node>
<node id="n6">
<data key="d6">
<y:ShapeNode>
<y:Geometry height="67.0" width="176.0" x="838.234375" y="291.5"/>
<y:Fill color="#4CAF50" transparent="false"/>
<y:BorderStyle color="#000000" type="line" width="1.0"/>
<y:NodeLabel alignment="center" autoSizePolicy="content" fontFamily="Dialog" fontSize="12" fontStyle="plain" hasBackgroundColor="false" hasLineColor="false" height="17.96875" horizontalTextPosition="center" iconTextGap="4" modelName="custom" textColor="#000000" verticalTextPosition="bottom" visible="true" width="116.048828125" x="29.9755859375" xml:space="preserve" y="24.515625">Stream Processing<y:LabelModel><y:SmartNodeLabelModel distance="4.0"/></y:LabelModel><y:ModelParameter><y:SmartNodeLabelModelParameter labelRatioX="0.0" labelRatioY="0.0" nodeRatioX="0.0" nodeRatioY="0.0" offsetX="0.0" offsetY="0.0" upX="0.0" upY="-1.0"/></y:ModelParameter></y:NodeLabel>
<y:Shape type="roundrectangle"/>
</y:ShapeNode>
</data>
</node>
<edge id="e0" source="n0" target="n1">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="331.30859375" y="325.0"/>
</y:Path>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e1" source="n0" target="n2">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="154.5" y="371.0"/>
</y:Path>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e2" source="n1" target="n3">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="white_delta"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e3" source="n0" target="n4">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e4" source="n5" target="n4">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e5" source="n5" target="n1">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="0.0" ty="0.0">
<y:Point x="798.69140625" y="325.0"/>
</y:Path>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
<edge id="e6" source="n6" target="n1">
<data key="d9"/>
<data key="d10">
<y:PolyLineEdge>
<y:Path sx="0.0" sy="0.0" tx="62.0" ty="0.0"/>
<y:LineStyle color="#000000" type="line" width="1.0"/>
<y:Arrows source="none" target="standard"/>
<y:BendStyle smoothed="false"/>
</y:PolyLineEdge>
</data>
</edge>
</graph>
<data key="d7">
<y:Resources/>
</data>
</graphml>
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment