flink and edge

common/docs/knowledgeBase/pulsar-flink-admin.txt

+PULSAR - FLINK ADMIN:
+
+PULSAR:
+This is a pub/sub msg queue (bus) where domains and stream processing jobs communicates via topics
+Topic structure (url) is:
+[durability://][tenant]/[namespace]/[topic]
+e.g: persistent://mcx/activities/activity
+Domains topics:
+- all our topics are under 'mcx' tenant
+- every domain has it's own namespace according to it's service name (default behavior)
+  e.g activities domain namespace is 'activities'
+- the topics are the api's e.g /activity api will have the topic url: 'persistent://mcx/activities/activity'
+- every domain creates it's own topic on startup, except activities that also creates the 'public' topics
+Streaming jobs:
+- all the streaming jobs have topics under 'public' namespace in the 'mcx' tenant
+  e.g. the cep-activities job subscribe to the topics: 'persistent://mcx/public/activities' and 'persistent://mcx/public/activity'
+Docker:
+- in the compose file under pulsar service
+- there are 3 ports to open/map: 
+  - 80 for web interface ( only for pulsar-standalone or pulsar-dashboard)
+  - 8080 for admin
+  - 6650 the actual pubsub port
+Web Interface:
+- There is an internal web interface if you run pulsar-standalone docker, but it is not 
+  showing the actual topics in real-time
+- pulsar-express:
+  https://github.com/bbonnin/pulsar-express 
+  This is a remote interface that connects to the pulsar (like the db tools for mongo/redis etc')
+  you can install it using:
+  npm install pulsar-express -g (add sudo if you're not on root)
+  when creating connections use the admin port mapping
+
+FLINK:
+This is a stream processing framework, is receives data from 'sources' (subscribes to topics),
+receive the msgs and process them in real time, and outputs (if there is output) to 'sinks' ( like http or public to topics)
+Every processing unit is called a JOB, each job has several tasks that runs on cluster of nodes.
+- the flink framework has a job manager and a task manager that has one or more nodes
+- the jobs are loaded to the flink from web interface/commandline or from pre-loaded directory
+- our jobs are published to the archiva so that they can be downloaded anywhere and uploaded to
+  the flink framework, the path is http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/[job-name]/[version]/[job-artifact].jar
+  e.g: http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/activities-cep-job/1.0.0/activities-cep-job-1.0.0-all.jar
+- the jobs doesn't create topics on pulsar, so the activities service must be up before uploading the job, and the pulsar
+  service must be up before starting the activities service
+Docker:
+- there are to docker services, one for job manager and one for task manager
+- port mapping 8081 for web interface
+- if the flink is running on a different network/machine then we need to define the domains needed by the jobs
+  in the extra_hosts 
+- since the jobs runs under the flink instances, the jobs logs is sent to the consule log which will appear in the instances docker log.
+Web Interface:
+- in port 8081, under 'Submit new Job' upload the job, click on it and click 'Submit'
+- it may fail on first try due to timeouts, try again before searching for errors in the logs
+

docker/composers/mcx-cep.yml

@@ -11,7 +11,8 @@ services:
     environment:
       - JOB_MANAGER_RPC_ADDRESS=jobmanager
     extra_hosts:
-      - "pulsar:172.16.1.244"
+      - "alerts:172.16.1.244"
+      - "scp:172.16.1.72"
       # volumes:
     #   - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
     networks:
@@ -31,22 +32,23 @@ services:
     environment:
       - JOB_MANAGER_RPC_ADDRESS=jobmanager
     extra_hosts:
-      - "pulsar:172.16.1.244"
+      - "alerts:172.16.1.244"
+      - "scp:172.16.1.72"
     # volumes:
     #   - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
     networks:
       - backend
 
-#   pulsar:
-#     image: apachepulsar/pulsar-standalone:2.4.2
-#     ports:
-#       - 8080:8080
-#       - 8082:80
-#       - 6650:6650
-# #    volumes:
-# #      - "/ext/pulsar/data:/pulsar/data"
-#     networks:
-#       - backend
+  pulsar:
+    image: apachepulsar/pulsar-standalone:2.4.2
+    ports:
+      - 8080:8080
+      - 8082:80
+      - 6650:6650
+#   volumes:
+#     - "/ext/pulsar/data:/pulsar/data"
+    networks:
+      - backend
 networks:
   backend:
     driver: bridge