Skip to content
Toggle navigation
P
Projects
G
Groups
S
Snippets
Help
ipgallery
/
devops
This project
Loading...
Sign in
Toggle navigation
Go to a project
Project
Repository
Registry
Issues
0
Merge Requests
0
Wiki
Settings
Activity
Graph
Charts
Create a new issue
Commits
Issue Boards
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Commit
d5896d1b
authored
Jan 26, 2020
by
Amir Aharon
Browse files
Options
_('Browse Files')
Download
Email Patches
Plain Diff
flink and edge
parent
0e84d0a8
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
67 additions
and
12 deletions
common/docs/knowledgeBase/pulsar-flink-admin.txt
docker/composers/mcx-cep.yml
products/MCX/design/Edge/DataFramwork.graphml
common/docs/knowledgeBase/pulsar-flink-admin.txt
0 → 100644
View file @
d5896d1b
PULSAR - FLINK ADMIN:
PULSAR:
This is a pub/sub msg queue (bus) where domains and stream processing jobs communicates via topics
Topic structure (url) is:
[durability://][tenant]/[namespace]/[topic]
e.g: persistent://mcx/activities/activity
Domains topics:
- all our topics are under 'mcx' tenant
- every domain has it's own namespace according to it's service name (default behavior)
e.g activities domain namespace is 'activities'
- the topics are the api's e.g /activity api will have the topic url: 'persistent://mcx/activities/activity'
- every domain creates it's own topic on startup, except activities that also creates the 'public' topics
Streaming jobs:
- all the streaming jobs have topics under 'public' namespace in the 'mcx' tenant
e.g. the cep-activities job subscribe to the topics: 'persistent://mcx/public/activities' and 'persistent://mcx/public/activity'
Docker:
- in the compose file under pulsar service
- there are 3 ports to open/map:
- 80 for web interface ( only for pulsar-standalone or pulsar-dashboard)
- 8080 for admin
- 6650 the actual pubsub port
Web Interface:
- There is an internal web interface if you run pulsar-standalone docker, but it is not
showing the actual topics in real-time
- pulsar-express:
https://github.com/bbonnin/pulsar-express
This is a remote interface that connects to the pulsar (like the db tools for mongo/redis etc')
you can install it using:
npm install pulsar-express -g (add sudo if you're not on root)
when creating connections use the admin port mapping
FLINK:
This is a stream processing framework, is receives data from 'sources' (subscribes to topics),
receive the msgs and process them in real time, and outputs (if there is output) to 'sinks' ( like http or public to topics)
Every processing unit is called a JOB, each job has several tasks that runs on cluster of nodes.
- the flink framework has a job manager and a task manager that has one or more nodes
- the jobs are loaded to the flink from web interface/commandline or from pre-loaded directory
- our jobs are published to the archiva so that they can be downloaded anywhere and uploaded to
the flink framework, the path is http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/[job-name]/[version]/[job-artifact].jar
e.g: http://municipalitybank.com:8081/repository/internal/com/ipgallery/esp/activities-cep-job/1.0.0/activities-cep-job-1.0.0-all.jar
- the jobs doesn't create topics on pulsar, so the activities service must be up before uploading the job, and the pulsar
service must be up before starting the activities service
Docker:
- there are to docker services, one for job manager and one for task manager
- port mapping 8081 for web interface
- if the flink is running on a different network/machine then we need to define the domains needed by the jobs
in the extra_hosts
- since the jobs runs under the flink instances, the jobs logs is sent to the consule log which will appear in the instances docker log.
Web Interface:
- in port 8081, under 'Submit new Job' upload the job, click on it and click 'Submit'
- it may fail on first try due to timeouts, try again before searching for errors in the logs
docker/composers/mcx-cep.yml
View file @
d5896d1b
...
...
@@ -11,7 +11,8 @@ services:
environment
:
-
JOB_MANAGER_RPC_ADDRESS=jobmanager
extra_hosts
:
-
"
pulsar:172.16.1.244"
-
"
alerts:172.16.1.244"
-
"
scp:172.16.1.72"
# volumes:
# - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
networks
:
...
...
@@ -31,22 +32,23 @@ services:
environment
:
-
JOB_MANAGER_RPC_ADDRESS=jobmanager
extra_hosts
:
-
"
pulsar:172.16.1.244"
-
"
alerts:172.16.1.244"
-
"
scp:172.16.1.72"
# volumes:
# - "/opt/flink/conf/flink-conf.yaml:/home/amir/git/devops/docker/composers/flink-conf.yaml"
networks
:
-
backend
#
pulsar:
#
image: apachepulsar/pulsar-standalone:2.4.2
#
ports:
#
- 8080:8080
#
- 8082:80
#
- 6650:6650
#
#
volumes:
#
#
- "/ext/pulsar/data:/pulsar/data"
#
networks:
#
- backend
pulsar
:
image
:
apachepulsar/pulsar-standalone:2.4.2
ports
:
-
8080:8080
-
8082:80
-
6650:6650
# volumes:
# - "/ext/pulsar/data:/pulsar/data"
networks
:
-
backend
networks
:
backend
:
driver
:
bridge
products/MCX/design/Edge/DataFramwork.graphml
0 → 100644
View file @
d5896d1b
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment