Provide documented example code for all database access methods supported by Pervasive.SQL on all platforms using all popular languages. Preferably useful for both beginner and advanced user as a reference guide.

See:

• Rosetta Stone
• Rosetta Code

For many years it struck me that code/coding examples were scarce. Also they varied over time (platforms, languages supported), but most of all stuck in time. Not very appealing for a starter, whether (s)he would be new to a programming language or to Pervasive.SQL.
Over the years I developed ideas on how to improve this and made some efforts writing code.
The task ahead is quite extensive. Especially if one wants to do a proper job.
Ideas change, new projects or tasks got in between, etc. Long story short it took some time and the result is very different than at first anticipated as my first idea was to write a single reference application which could later be ported to other languages/platforms.

Based on the paragraph Database Access Methods in the Actian Pervasive.SQL V13 online documentation I created a Bash shellscript (mk_dirs.sh), taking a single argument being the programminglanguage name, which creates a directory structure listing all the database access methods as subdirectories. By using this script I was forced to look into and document all(?!) possibilities regardless how odd. All subdirectories contain their own markdown ReadMe file describing the (im)possibilities and code if provided.
All programminglanguages have a ReadMe markdown file in their root directory describing the ins and outs, what is and isn’t implemented as well as a Results markdown file to register what has been tested on which platform.

The goals was not to infringe any copyrights, so headers must be copied from SDKs which can be downloaded from the Actian website. The same goes for example code which can be copy/pasted from the website. It would be great if example code (& headers) could be made available from a repository.
When looking around on Github one can find copyrighted header files. I leave it to Actian to add them.

I very much welcome improvements, comments and other contributions.
Personally I can think of a view:

• All code should confirm to coding standards.
• Refactoring/cleanup of code.
• All code should be very rich in comments. Annotate all database calls.
• All code should be made very defensive: if an error occurs it should be reported or at least logged.
• All code should be properly tested. Preferably on all relevant platforms. Which on turn should be documented.
• Code must be written or adapted for other platforms. Notably: Mac, IOS, Android
• Some obvious languages/platforms are missing. Notably: Win/VS: c#, VB.net, Win/Embarcadero C++, Win/MingW or other GNU C/C++, IOS/Objective C, Android/Java, Mac/making the bash-shell scripts compatible/supportive.
• Also some languages which used to be supported/were important do not have sample code yet. What springs to mind: Cobol, Delphi, … ? And some are no longer important: (Visual) Basic (pre .net), Pascal, Turboc (DOS), Watcom C/C++ (DOS)
• Some ‘languages’ are not very demonstratable as they seem to require severe boilerplating, project management and/or integration in an IDE. ASP.NET being an example.
• Integrated platforms are not listed. For example Magic It probably makes no sense in listing them. Other platforms used in the past: Clarion and Power Builder
• Another subject which requires attention is web-based development. One can think of: Windows/ASP, Python/Flask, Python/Django, Ruby/Ruby-on-rails and Javascript, NodeJS. Optionally expanded by new kids on the block such as Dart/Flutter, Meteor, etc. although a lot of them are based on Javascript.
• Drivers. Currently especially one springs to mind: SQLAlchemy-Pervasive : it needs some serious TLC.
• Currently a strong focus is on database connectivity.
  Ultimately an application supporting commandline, curses (TUI), GUI while using all calls available in APIs (Btrieve, Btrieve2, ODBC, JDBC) would be a real bonus. It would cater for demoing, illustrating how calls should be used and obviously would provide a great test, especially if the code could be run using test automation.
  This would be a major thing to design and implement properly. Some baby steps in this process alone would be great.

I am fully aware that most code does not comply to above standards. Refactoring all code would take a lot of time which would pospone the initial release or maybe even prevent it.
For this reason I am releasing code which does not meet my views on proper coding.

See the Credits.md file. This file applies to the entire project.

See the License.md file. This file applies to the entire project.

For sake of completeness and uniformity all access methods mentioned in the programmers manual are listed as options for all languages. The combinations can be quite absurd or exotic. Obviously especially those are not implemented (yet) and/or properly tested.
All code and documentation in this repository is provided as is.
By no means I am an expert in all languages provided. The goal is to at least deliver working code which is a very low standard, but not uncommon unfortunately. Writing about programming versus Software Engineering can fill up bookshelves. Lets no go there now.
Hopefully the quality of code will increase over time if people being expert in a certain language participate and improve code.
Most code is tested on Linux only unless stated otherwise. To improve maturity and clearity on this subject test result tables have been added.

Runs Cypress end-to-end tests on Netlify Build

You can install this plugin in the Netlify UI from this direct in-app installation link or from the Plugins directory.

For file based installation, add netlify-plugin-cypress NPM package as a dev dependency to your repository.

npm install --save-dev netlify-plugin-cypress
# or
yarn add -D netlify-plugin-cypress

And then add the plugin’s name to the list of build plugins in netlify.toml file as shown in the examples below.

note: this plugin assumes you have already installed Cypress as a dev NPM dependency.

This plugin installs via Puppeteer Chromium browser, which is also cached inside ./node_modules folder.

When Netlify Build system runs it performs 2 steps essentially:

1. builds the site
2. deploys the site

Every plugin that wants to perform some actions can do so before the build, after the build (but before the deploy), and after the deploy. The Netlify uses the following names for these events

"preBuild"
1. builds the site
"postBuild"
2. deploys the site
"onSuccess"
"onFailure"

Thus every plugin can register itself to be executed before a site is built using “preBuild” event, or after a successful deploy using “onSuccess” event name, etc.

This plugin netlify-plugin-cypress by default runs during the “onSuccess” event, testing the deployed site. The Netlify Build system gives the URL to the plugin and it runs Cypress against that URL using the Cypress NPM module API.

Optionally, you can also run tests during “preBuild” and “postBuild” steps. This is useful if you want to ensure the site is working even before deploying it to Netlify servers. Finally, this plugin does not use “onFailure” event which happens only if Netlify fails to deploy the site.

Running Cypress tests by default uses the “onSuccess” step of the build pipeline. By this point Netlify has already deployed the site. Even if the tests fail now, the Netlify shows the successful deployment – the site is live! To really prevent the broken deploys, we suggest using Cypress GitHub / GitLab / Bitbucket integration to fail the status checks on a pull request.

We also suggest running tests during the “preBuild” and/or “postBuild” steps. If the tests fail during these steps, the Netlify fails the entire build and does not deploy the broken site.

Finally, you can set up Slack notifications on failed tests against the deployed site. At least you will quickly find out if the deployed site fails the E2E tests and would be able to roll back the deploy.

Here is the most basic Netlify config file netlify.toml with just the Cypress plugin

[[plugins]]
  # runs Cypress tests against the deployed URL
  package = "netlify-plugin-cypress"

The example file above should be enough to run Cypress tests in any existing Netlify project.

We strongly recommend setting CYPRESS_CACHE_FOLDER to place the Cypress binary inside the node_modules folder to cache it between builds

# explicit commands for building the site
# and the folder to publish
[build]
command = "npm run build"
publish = "build"

[build.environment]
# cache Cypress binary in local "node_modules" folder
# so Netlify caches it
CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
# set TERM variable for terminal output
TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"

See netlify-plugin-cypress-example repo.

Typescript users may need to add a install before the build command. For a yarn user with a typescript app, the build section of the Netlify configuration might look like this:

[build]
command = "yarn install && yarn build"
publish = "build"

# ...remaining configuration...

Read the full tutorial at Test Sites Deployed To Netlify Using netlify-plugin-cypress.

Note: if any tests against the deployed URL fail, the Netlify build still considers it a success. Thus if you want to have a test check against the deploy, install Cypress GitHub App. The app will provide its own failing status check in this case.

You can control the browser, the specs to run, record tests on Cypress Dashboard, etc, see manifest.yml file.

To record test results and artifacts on Cypress Dashboard, set record: true plugin input and set CYPRESS_RECORD_KEY as an environment variable via Netlify Deploy settings.

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true

See cypress-example-kitchensink and recorded results at Cypress Dashboard

Security note 🔐: you should keep your CYPRESS_RECORD_KEY secret. You can control how Netlify builds external pull requests, see the doc – you never want to expose sensitive environment variables to outside builds.

If you are recording test results to Cypress Dashboard, you should also install Cypress GitHub Integration App to see status checks from individual groups or from individual specs per commit. See netlify-plugin-prebuild-example PR #8 pull request for an example.

You can change the group name for the recorded run using group parameter

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"

You can give recorded run tags using a comma-separated string. If the tag is not specified, Netlify context will be used (production, deploy-preview or branch-deploy)

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"
  tag = "nightly,production"

Run only a single spec or specs matching a wildcard

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  spec = "cypress/integration/smoke*.js"

See cypress-example-kitchensink for instance.

By default all tests run using the Chromium browser. If you want to use Electron:

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # allowed values: electron, chromium
  browser = "electron"

If you would like to use a different Cypress config file instead of cypress.json, specify it using the configFile option

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  configFile = "cypress.netlify.config.js"

SPAs need catch-all redirect setup to make non-root paths accessible by tests. You can enable this with spa parameter.

[[plugins]]
# local Cypress plugin will test our site after it is built
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # can also use "spa = true" to use "index.html" by default
  spa = "index.html"

See lws-spa for more options and tests/routing example.

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests against the local development server. You can start the local server, wait for it to respond and then run Cypress tests by passing parameters to this plugin. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against development server
  # before building it (and testing the built site)
  [plugins.inputs.preBuild]
    enable = true
    start = 'npm start'
    wait-on = 'http://localhost:3000'
    wait-on-timeout = '30' # seconds

Parameters you can place into preBuild inputs: start, wait-on, wait-on-timeout, spec, record, group, and tag.

See netlify-plugin-prebuild-example repo

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests locally after building the static site. Cypress includes a local static server for this case but you can specify your own command if needed by using the start argument. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true

Parameters you can place into postBuild inputs: spec, record, group, tag, start and spa.

If your site requires all unknown URLs to redirect back to the index page, use the spa parameter

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true
    # must allow our test server to redirect unknown routes to "https://github.com/"
    # so that client-side routing can correctly route them
    # can be set to true or "index.html" (or similar fallback filename in the built folder)
    spa = true
    start = 'npm start'

See the routing example.

Even better when testing the prebuilt site is to run the Netlify CLI to make sure the local API redirects and Netlify functions work in addition to the web site. Add netlify-cli as a dev dependency and start it during testing.

$ npm i -D netlify-cli

[[plugins]]
  package = "netlify-plugin-cypress"
  # start Netlify server
  [plugins.inputs.preBuild]
    start = 'npx netlify dev'
    wait-on = 'http://localhost:8888'

For more, see tests/test-netlify-dev example and read Testing Netlify Function section.

If you are testing the site before building it and want to skip testing the deployed URL

[[plugins]]
  package = "netlify-plugin-cypress"
  # do not test the deployed URL
  [plugins.inputs]
    enable = false
  # test the local site
  [plugins.inputs.preBuild]
    enable = true

Running tests in parallel is not supported because Netlify plugin system runs on a single machine. Thus you can record the tests on Cypress Dashboard, but not run tests in parallel. If Netlify expands its build offering by allowing multiple build machines, we could take advantage of it and run tests in parallel.

When serving the built folder, we automatically serve .html files. For example, if your folder has the following structure:

public/
  index.html
  pages/
    about.html

The public folder is served automatically and the following test successfully visits both the root and the about.html pages:

cy.visit("https://github.com/")
cy.visit('/pages/about') // visits the about.html

• The default browser has been switched to Chromium. If you want to use the built-in Electron use an explicit option browser
• We have changed the default testing phase. In v1 the tests executed after building the site by default. In v2 the tests run against the deployed URL by default, and you need to enable the testing during preBuild or postBuild steps.

Set environment variable DEBUG=netlify-plugin-cypress to see the debug logs. To see even more information, set DEBUG=netlify-plugin-cypress,netlify-plugin-cypress:verbose

Warning: be careful with verbose logging, since it can print all environment variables passed to the plugin, including tokens, API keys, and other secrets.

This project is licensed under the terms of the MIT license.

Read the contributing guide

Runs Cypress end-to-end tests on Netlify Build

You can install this plugin in the Netlify UI from this direct in-app installation link or from the Plugins directory.

For file based installation, add netlify-plugin-cypress NPM package as a dev dependency to your repository.

npm install --save-dev netlify-plugin-cypress
# or
yarn add -D netlify-plugin-cypress

And then add the plugin’s name to the list of build plugins in netlify.toml file as shown in the examples below.

note: this plugin assumes you have already installed Cypress as a dev NPM dependency.

This plugin installs via Puppeteer Chromium browser, which is also cached inside ./node_modules folder.

When Netlify Build system runs it performs 2 steps essentially:

1. builds the site
2. deploys the site

Every plugin that wants to perform some actions can do so before the build, after the build (but before the deploy), and after the deploy. The Netlify uses the following names for these events

"preBuild"
1. builds the site
"postBuild"
2. deploys the site
"onSuccess"
"onFailure"

Thus every plugin can register itself to be executed before a site is built using “preBuild” event, or after a successful deploy using “onSuccess” event name, etc.

This plugin netlify-plugin-cypress by default runs during the “onSuccess” event, testing the deployed site. The Netlify Build system gives the URL to the plugin and it runs Cypress against that URL using the Cypress NPM module API.

Optionally, you can also run tests during “preBuild” and “postBuild” steps. This is useful if you want to ensure the site is working even before deploying it to Netlify servers. Finally, this plugin does not use “onFailure” event which happens only if Netlify fails to deploy the site.

Running Cypress tests by default uses the “onSuccess” step of the build pipeline. By this point Netlify has already deployed the site. Even if the tests fail now, the Netlify shows the successful deployment – the site is live! To really prevent the broken deploys, we suggest using Cypress GitHub / GitLab / Bitbucket integration to fail the status checks on a pull request.

We also suggest running tests during the “preBuild” and/or “postBuild” steps. If the tests fail during these steps, the Netlify fails the entire build and does not deploy the broken site.

Finally, you can set up Slack notifications on failed tests against the deployed site. At least you will quickly find out if the deployed site fails the E2E tests and would be able to roll back the deploy.

Here is the most basic Netlify config file netlify.toml with just the Cypress plugin

[[plugins]]
  # runs Cypress tests against the deployed URL
  package = "netlify-plugin-cypress"

The example file above should be enough to run Cypress tests in any existing Netlify project.

We strongly recommend setting CYPRESS_CACHE_FOLDER to place the Cypress binary inside the node_modules folder to cache it between builds

# explicit commands for building the site
# and the folder to publish
[build]
command = "npm run build"
publish = "build"

[build.environment]
# cache Cypress binary in local "node_modules" folder
# so Netlify caches it
CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
# set TERM variable for terminal output
TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"

See netlify-plugin-cypress-example repo.

Typescript users may need to add a install before the build command. For a yarn user with a typescript app, the build section of the Netlify configuration might look like this:

[build]
command = "yarn install && yarn build"
publish = "build"

# ...remaining configuration...

Read the full tutorial at Test Sites Deployed To Netlify Using netlify-plugin-cypress.

Note: if any tests against the deployed URL fail, the Netlify build still considers it a success. Thus if you want to have a test check against the deploy, install Cypress GitHub App. The app will provide its own failing status check in this case.

You can control the browser, the specs to run, record tests on Cypress Dashboard, etc, see manifest.yml file.

To record test results and artifacts on Cypress Dashboard, set record: true plugin input and set CYPRESS_RECORD_KEY as an environment variable via Netlify Deploy settings.

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true

See cypress-example-kitchensink and recorded results at Cypress Dashboard

Security note 🔐: you should keep your CYPRESS_RECORD_KEY secret. You can control how Netlify builds external pull requests, see the doc – you never want to expose sensitive environment variables to outside builds.

If you are recording test results to Cypress Dashboard, you should also install Cypress GitHub Integration App to see status checks from individual groups or from individual specs per commit. See netlify-plugin-prebuild-example PR #8 pull request for an example.

You can change the group name for the recorded run using group parameter

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"

You can give recorded run tags using a comma-separated string. If the tag is not specified, Netlify context will be used (production, deploy-preview or branch-deploy)

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"
  tag = "nightly,production"

Run only a single spec or specs matching a wildcard

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  spec = "cypress/integration/smoke*.js"

See cypress-example-kitchensink for instance.

By default all tests run using the Chromium browser. If you want to use Electron:

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # allowed values: electron, chromium
  browser = "electron"

If you would like to use a different Cypress config file instead of cypress.json, specify it using the configFile option

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  configFile = "cypress.netlify.config.js"

SPAs need catch-all redirect setup to make non-root paths accessible by tests. You can enable this with spa parameter.

[[plugins]]
# local Cypress plugin will test our site after it is built
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # can also use "spa = true" to use "index.html" by default
  spa = "index.html"

See lws-spa for more options and tests/routing example.

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests against the local development server. You can start the local server, wait for it to respond and then run Cypress tests by passing parameters to this plugin. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against development server
  # before building it (and testing the built site)
  [plugins.inputs.preBuild]
    enable = true
    start = 'npm start'
    wait-on = 'http://localhost:3000'
    wait-on-timeout = '30' # seconds

Parameters you can place into preBuild inputs: start, wait-on, wait-on-timeout, spec, record, group, and tag.

See netlify-plugin-prebuild-example repo

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests locally after building the static site. Cypress includes a local static server for this case but you can specify your own command if needed by using the start argument. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true

Parameters you can place into postBuild inputs: spec, record, group, tag, start and spa.

If your site requires all unknown URLs to redirect back to the index page, use the spa parameter

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true
    # must allow our test server to redirect unknown routes to "https://github.com/"
    # so that client-side routing can correctly route them
    # can be set to true or "index.html" (or similar fallback filename in the built folder)
    spa = true
    start = 'npm start'

See the routing example.

Even better when testing the prebuilt site is to run the Netlify CLI to make sure the local API redirects and Netlify functions work in addition to the web site. Add netlify-cli as a dev dependency and start it during testing.

$ npm i -D netlify-cli

[[plugins]]
  package = "netlify-plugin-cypress"
  # start Netlify server
  [plugins.inputs.preBuild]
    start = 'npx netlify dev'
    wait-on = 'http://localhost:8888'

For more, see tests/test-netlify-dev example and read Testing Netlify Function section.

If you are testing the site before building it and want to skip testing the deployed URL

[[plugins]]
  package = "netlify-plugin-cypress"
  # do not test the deployed URL
  [plugins.inputs]
    enable = false
  # test the local site
  [plugins.inputs.preBuild]
    enable = true

Running tests in parallel is not supported because Netlify plugin system runs on a single machine. Thus you can record the tests on Cypress Dashboard, but not run tests in parallel. If Netlify expands its build offering by allowing multiple build machines, we could take advantage of it and run tests in parallel.

When serving the built folder, we automatically serve .html files. For example, if your folder has the following structure:

public/
  index.html
  pages/
    about.html

The public folder is served automatically and the following test successfully visits both the root and the about.html pages:

cy.visit("https://github.com/")
cy.visit('/pages/about') // visits the about.html

• The default browser has been switched to Chromium. If you want to use the built-in Electron use an explicit option browser
• We have changed the default testing phase. In v1 the tests executed after building the site by default. In v2 the tests run against the deployed URL by default, and you need to enable the testing during preBuild or postBuild steps.

Set environment variable DEBUG=netlify-plugin-cypress to see the debug logs. To see even more information, set DEBUG=netlify-plugin-cypress,netlify-plugin-cypress:verbose

Warning: be careful with verbose logging, since it can print all environment variables passed to the plugin, including tokens, API keys, and other secrets.

This project is licensed under the terms of the MIT license.

Read the contributing guide

It’s an early alpha version of the platform for SONM.io project.

For now it has lots of unfinished task. The main idea is to show that such platform can be implemented and to chose a techstack for future implementation.

This repository contains code for Hub, Miner and CLI.

A docker container contained every CLI, Miner, Hub can be found on public DockerHub: sonm/insonmnia

docker pull sonm/insonmnia

If you want it’s easy to build all the components. You need golang > 1.8:

make build

Also there is a Dockerfile to build a container:

docker build .

Look at milestone https://github.com/sonm-io/insonmnia/milestones

To start a hub it’s needed to expose a couple of ports. 10001 handles gRCP requests from CLI 10002 is used to handle communication with miners

docker run --rm -p 10002:10002 -p 10001:10001  sonm/insonmnia sonmhub

To run Miner from the container you need to pass docker socket inside and specify IP of the HUB

  docker run --net host -e DOCKER_API_VERSION=1.24 -v /run:/var/run sonm/insonmnia:alpha3 sonmminer -h <hubip:10002>

CLI sends commands to a hub. A hub must be pointed via –hub=hubip:port. Port is usually 10001.

Just check that a hub is reachable and alive.

sonmcli --hub <hubip:10001> ping
OK

List shows a list of miners connected to a hub with tasks assigned to them.

NOTE: later each miner will have a unique signed ID instead of host:port

sonmcli --hub <hubip:port> list
Connected Miners
{
	"<minerip:port": {
		"values": [
			"2b845fcc-143a-400b-92c7-aac2867ab62f",
			"412dd411-96df-442a-a397-6a2eba9147f9"
		]
	}
}

To start a container you have to pick a hub and miner connected to that hub. You can pick a miner from output of List command. See above.

./sonmcli --hub <hubip:port> --timeout=3000s  start --image schturmfogel/sonm-q3:alpha  --miner=<minerhost:port>

The result would look like:

ID <jobid>, Endpoint [27960/tcp-><ip:port> 27960/udp-><ip:port>]

• jobid is an unique name for the task. Later it can be used to specify a task for various operations.
• Endpoint describes mapping of exposed ports (google for Docker EXPOSE) to the real ports of a miner

NOTE: later STUN will be used for UDP packets and LVS (ipvs) or userspace proxy (like SSH tunnel) for TCP. Miners who have a public IPv4 or can be reached via IPv6 would not need this proxy. The proxy is intended to get through NAT.

To stop the task just provide the jobid

sonmcli --hub <hubip:port> stop <jobid>

Dockerfile for the image should follow several requirements:

• ENTRYPOINT or CMD or both must present
• Network ports should be specified via EXPOSE

Technologies we use right now:

• golang is the main language. Athough golang has disadvantages, we believe that its model is good for fast start and the code is easy to understand. The simplicity leads to less errors. Also it makes easy to contribute to a project as a review process is very clean.
• Docker is a heart of an isolation in our platform. We rely on security features (It’s not 100% safe!), metrics, ecosystem it provides. The cool thing Docker is supported by many platforms. Also Docker works a lot on a unikernel approach for container based applications, which opens a huge field for security and portability improvements.
• whisper as a discovery protocol
• Until the epoch of IPv6 begins we should bring a way to get through NAT. The solution depends on a concrete transport layer. For example, different approaches should be used for UDP (e.g. STUN) and TCP (naive userspace proxy). Each approach has its own overhead and the best fit solution depends on a task.
• gRPC is an API protocol between components. It’s very easy to extend, supports traffic compression, flexible auth model, supported by many language. It’s becoming more and more popular as a technology for RPC.

Hub provides public gRPC-based API. proto files can be found in proto dir.

Miner is expected to discover a Hub using Whisper. Later the miner connects to the hub via TCP. Right now a Miner must have a public IP address. Hub sends orders to the miner via gRPC on top of the connection. Hub pings the miner from time to time.

It’s an early alpha version of the platform for SONM.io project.

For now it has lots of unfinished task. The main idea is to show that such platform can be implemented and to chose a techstack for future implementation.

This repository contains code for Hub, Miner and CLI.

A docker container contained every CLI, Miner, Hub can be found on public DockerHub: sonm/insonmnia

docker pull sonm/insonmnia

If you want it’s easy to build all the components. You need golang > 1.8:

make build

Also there is a Dockerfile to build a container:

docker build .

Look at milestone https://github.com/sonm-io/insonmnia/milestones

To start a hub it’s needed to expose a couple of ports. 10001 handles gRCP requests from CLI 10002 is used to handle communication with miners

docker run --rm -p 10002:10002 -p 10001:10001  sonm/insonmnia sonmhub

To run Miner from the container you need to pass docker socket inside and specify IP of the HUB

  docker run --net host -e DOCKER_API_VERSION=1.24 -v /run:/var/run sonm/insonmnia:alpha3 sonmminer -h <hubip:10002>

CLI sends commands to a hub. A hub must be pointed via –hub=hubip:port. Port is usually 10001.

Just check that a hub is reachable and alive.

sonmcli --hub <hubip:10001> ping
OK

List shows a list of miners connected to a hub with tasks assigned to them.

NOTE: later each miner will have a unique signed ID instead of host:port

sonmcli --hub <hubip:port> list
Connected Miners
{
	"<minerip:port": {
		"values": [
			"2b845fcc-143a-400b-92c7-aac2867ab62f",
			"412dd411-96df-442a-a397-6a2eba9147f9"
		]
	}
}

To start a container you have to pick a hub and miner connected to that hub. You can pick a miner from output of List command. See above.

./sonmcli --hub <hubip:port> --timeout=3000s  start --image schturmfogel/sonm-q3:alpha  --miner=<minerhost:port>

The result would look like:

ID <jobid>, Endpoint [27960/tcp-><ip:port> 27960/udp-><ip:port>]

• jobid is an unique name for the task. Later it can be used to specify a task for various operations.
• Endpoint describes mapping of exposed ports (google for Docker EXPOSE) to the real ports of a miner

NOTE: later STUN will be used for UDP packets and LVS (ipvs) or userspace proxy (like SSH tunnel) for TCP. Miners who have a public IPv4 or can be reached via IPv6 would not need this proxy. The proxy is intended to get through NAT.

To stop the task just provide the jobid

sonmcli --hub <hubip:port> stop <jobid>

Dockerfile for the image should follow several requirements:

• ENTRYPOINT or CMD or both must present
• Network ports should be specified via EXPOSE

Technologies we use right now:

• golang is the main language. Athough golang has disadvantages, we believe that its model is good for fast start and the code is easy to understand. The simplicity leads to less errors. Also it makes easy to contribute to a project as a review process is very clean.
• Docker is a heart of an isolation in our platform. We rely on security features (It’s not 100% safe!), metrics, ecosystem it provides. The cool thing Docker is supported by many platforms. Also Docker works a lot on a unikernel approach for container based applications, which opens a huge field for security and portability improvements.
• whisper as a discovery protocol
• Until the epoch of IPv6 begins we should bring a way to get through NAT. The solution depends on a concrete transport layer. For example, different approaches should be used for UDP (e.g. STUN) and TCP (naive userspace proxy). Each approach has its own overhead and the best fit solution depends on a task.
• gRPC is an API protocol between components. It’s very easy to extend, supports traffic compression, flexible auth model, supported by many language. It’s becoming more and more popular as a technology for RPC.

Hub provides public gRPC-based API. proto files can be found in proto dir.

Miner is expected to discover a Hub using Whisper. Later the miner connects to the hub via TCP. Right now a Miner must have a public IP address. Hub sends orders to the miner via gRPC on top of the connection. Hub pings the miner from time to time.

Use systemd & ssh magic, to dynamically punch through firewalls; with “scale to zero” because coolness.

I have a bunch of systems I need to talk to which are hidden behind multiple jumpboxes.
I want to use local clients to talk to APIs and systems behind those jumpboxes,
I don’t want to interactively log in to any jumpbox and run stuff or leave state there[1].
I want my ssh connections to be relatively short lived and don’t want to
manually manage them; they should be established and closed on demand.

In my case this is mostly to talk to things like

• BOSH directors
• opsmanagers
• credhubs
• TKGi APIs
• kube apiservers
• …

via socks5, where those live inside private networks accessible only via one ore
more bastion / jump hosts.

[1]: That should be forbidden anyway. No state and no interactive sessions on
jumpboxes. Don’t do it.

mkdir -p ~/.config/systemd/{autosocks,user}
cp -r units/* ~/.config/systemd/autosocks/
cd ~/.config/systemd/user

proxyhost='opsman.nonprod.acme'
proxyport=65500

proxyhost_esc="$(systemd-escape "$proxyhost")"

ln -s ../autosocks/autosocks-@.socket              "autosocks-${proxyhost_esc}@${proxyport}.socket"
ln -s ../autosocks/autosocks-@.service             "autosocks-${proxyhost_esc}@${proxyport}.service"
ln -s ../autosocks/autosocks-connection-@.service  "autosocks-connection-${proxyhost_esc}@${proxyport}.service"

systemctl --user daemon-reload
systemctl --user start "autosocks-${proxyhost_esc}@${proxyport}.socket"

I don’t care about which or how many jumpboxes I need to use. I set it up once
in my ~/.ssh/config and forget about it.

Host *
  ControlMaster auto
  ControlPath ~/.ssh/cm-%C
  ControlPersist no

Host *.prod.acme
  ForwardAgent yes
  ProxyJump bastion.prod.acme

Host *.sandbox.acme
  ForwardAgent yes
  ProxyJump bastion.sandbox.acme

This, and the fact that I use the ssh-agent, pull keys from vault and add them
to the ssh-agent with a limited lifetime (ssh-add -t ...) allows me to e.g.
do ssh opsman.prod.acme and it logs me.

Going forward, for each system you want to use to proxy through this needs to
be true: You need to be able to ssh in with just providing the hostname. There
must not be any other interaction (password prompt) or any other config needed;
everything needs to be handled by the ssh config and the ssh-agent.

Now the fun part: Setting up systemd to dynamically manage your connection

1. in units you’ll find 3 systemd unit templates. You need to copy
   those to your user’s systemd directory (e.g.
   ~/.config/systemd/autosocks/). Those are just templates and won’t do anything
   by themselves, but implement all the needed parts once they are
   instantiated.

   • autosocks-connection-@.service

     This implements the actual ssh connection, that will be started on-demand.
     Essentially, it will just call ssh $hostname and will use a specific
     port to bind ssh as a socks server on.

   • autosocks-@.service

     This will run systemd-socket-proxyd which will take get the socket from
     systemd once connections have been observed and will proxy to the the
     socks proxy created by ssh.

     It will also make sure to:

     • Start a autosocks-connection-@.service instance on-demand and shut it down
     • Shut itself down, once ther is no traffic going through

   • autosocks-@.socket

     this will have systemd listen on a port and wait for connections. Once
     a connection is observed, systemd will start a autosocks-@.service
     instance and pass the socket to that.

2. Instantiate & start those units

   systemd units can be templated with an “instance” (the thing after the
   @) and a “prefix” (the thing between the last - and the @).
   We’ll use that to configure the host we want to proxy through and the local
   port we want the socks service to listen on.

   With the hosts from the above ssh config we can do the following:

   cd ~/.config/systemd/user/
   
   # set up a proxy through opsman.sandbox.acme
   ln -s ../autosocks/autosocks-@.socket             autosocks-opsman.sandbox.acme@65500.socket
   ln -s ../autosocks/autosocks-@.service            autosocks-opsman.sandbox.acme@65500.service
   ln -s ../autosocks/autosocks-connection-@.service autosocks-connection-opsman.sandbox.acme@65500.service
   # enable the sandbox proxy
   systemd --user start autosocks-opsman.sandbox.acme@65500.socket
   
   # set up a proxy through opsman.prod.acme
   ln -s ../autosocks/autosocks-@.socket             autosocks-opsman.prod.acme@65501.socket
   ln -s ../autosocks/autosocks-@.service            autosocks-opsman.prod.acme@65501.service
   ln -s ../autosocks/autosocks-connection-@.service autosocks-connection-opsman.prod.acme@65501.service
   # enable the prod proxy
   systemd --user start autosocks-opsman.prod.acme@65501.socket

   This creates a listening socket on 127.0.0.1:65500 for sandbox and
   127.0.0.1:65501 for prod.

   If you did a netstat -lnp or similar, you’d see that on the ports 65500
   and 65501 systemd is listening on. But not much more.

   Note: If you want to connect to a host with a - in its FQDN/name, you
   need to escape that - by \x2d. So the commands to link the units for a
   hostname like opsman.my-bizunit.acme would become something like:

   ln -s ../autosocks/autosocks-@.socket             autosocks-opsman.my\\x2dbizunit.acme@65502.socket
   ln -s ../autosocks/autosocks-@.service            autosocks-opsman.my\\x2dbizunit.acme@65502.service
   ln -s ../autosocks/autosocks-connection-@.service autosocks-connection-opsman.my\\x2dbizunit.acme@65502.service

Now let’s use that thing!

# It takes a bit to start all the systems, but eventually this should return
# (given, the host you proxy through has internet access).
https_proxy=socks5://localhost:65500 curl https://google.com/

# This is faster, because the connection is already established
https_proxy=socks5://localhost:65500 curl https://google.com/

And what you see now:

• There is a ssh process, connecting to opsman.sandbox.acme which is listening as a socks server on 127.255.255.254:65500
• There is a process systemd-socket-proxyd running now, which proxies everything on port 127.0.0.1:65500 to port 127.255.255.254:65500

If you wait a bit and there is no further traffic going through 65500

• both ssh and systemd-socket-proxyd will shut down automatically.
• the socket listening on 127.0.0.1:65500 will be moved back to the systemd process

• Once the autosocks-@.socket instance is running, systemd will listen on
  that intance’s port (e.g. 127.0.0.1:65500)
• If there is a connection to that port, systemd starts
  • ssh to the unit’s instance’s host, running a socks proxy service on the
    same port but on 127.255.255.254 (e.g. 127.255.255.254:65500)
    • 127.255.255.254 is just another IP on the loopback interface. Any other
      local IP can be used; this is configured as AUTOSOCKS_IP in the
      autosocks-@.service & autosocks-connection-@.service units
    • It’s probably best to keep 127.255.255.254 exclusively for autosocks to
      reduce the chance of port collisions
  • systemd-socket-proxyd will be started; it will receive the socket from
    systemd and use it to receive traffic on 127.0.0.1:65500 and proxy it
    through to 127.255.255.254:65500
• systemd-socket-proxyd will shutdown, if it’s idle for a bit
  • once systemd-socket-proxyd shuts down, so will the ssh connection providing the socks server
• if any of either the ssh connection or the activation socket will shutdown or
  die, so will the systemd-socket-proxyd. The system will however try to
  reestablish all the things on the next connection attempt to the instnace’s
  port

• See all statuses: systemctl --user status --all 'autosocks*'
• Tail all logs: journalctl --user --all -f -u 'autosocks*'

• …

This is pretty much stolen from ankostis over at StackExchange. The
only thing I did was to make it a bit easier to reuse by making use of the
units’ “instance” and “prefix”.

Diagnose 14 pathologies on Chest X-Ray using Deep Learning. Perform diagnostic interpretation using GradCAM Method

This project is a complilation of several sub-projects from Coursera 3-course IA for Medical Specialization. The objective is to use a deep learning model to diagnose pathologies from Chest X-Rays.

The project uses a pretrained DenseNet-121 model able to diagnose 14 labels such as Cardiomegaly, Mass, Pneumothorax or Edema. In other words, this single model can provide binary classification predictions for each of the 14 labeled pathologies.

Weight normalization is performed to offset the low prevalence of the abnormalities among the dataset of X-Rays (class imbalance).

Finally the GradCAM technique is used to highlight and visualize where the model is looking, which area of interest is used to make the prediction. This is a tool which can be helpful for discovery of markers, error analysis, training and even in deployment.

The project uses chest x-ray images taken from the public ChestX-ray8 dataset. This dataset contains 108,948 frontal-view X-ray images of 32,717 unique patients. Each image in the data set contains multiple text-mined labels identifying 14 different pathological conditions. These in turn can be used by physicians to diagnose 8 different diseases. For the project we have been working with a ~1000 image subset of the images.

• 875 images to be used for training.
• 109 images to be used for validation.
• 420 images to be used for testing.

The dataset includes a CSV file that provides the ground truth labels for each X-ray.

DenseNet was introduced in 2017 in an award-winning paper by Gao Huang et al. 2018 called Densely Connected Convolutional Networks. The model was able to outperform previous architectures like ResNet (which I covered in a another project Skin Cancer AI dermatologist).

Regardless of the architectural designs of these networks, they all try to create channels for information to flow between the initial layers and the final layers. DenseNet, with the same objective, create paths between the layers of the network. Parts of this summary are can be found in this review.

• DenseNet key novelty: Densenet is a convolutional network where each layer is connected to all other layers that are deeper in the network
  • The first layer is connected to the 2nd, 3rd, 4th etc.
  • The second layer is connected to the 3rd, 4th, 5th etc.

Each layer in a dense block receives feature maps from all the preceding layers, and passes its output to all subsequent layers. Feature maps received from other layers are fused through concatenation, and not through summation (like in ResNets). Extracted feature maps are continuously added together with previous ones which avoids redundant and duplicate work.

This allows the network to re-use learned information and be more efficient. Such networks require fewer layers. State of the art results are achieved with as low as 12 channel feature maps. This also means the network has fewer parameters to learn and is therefore easier to train. Amongst all variants, DenseNet-121 is the standard one.

Key contributions of the DenseNet architecture:

• Alleviates vanishing gradient problem ( as networks get deeper, gradients aren’t back-propagated sufficiently to the initial layers of the network. The gradients keep getting smaller as they move backwards into the network and as a result, the initial layers lose their capacity to learn the basic low-level features)
• Stronger feature propagation
• Feature re-use
• Reduced parameter count

DenseNet is composed of Dense blocks. In those blocks, the layers are densely connected together: Each layer receive in input all previous layers output feature maps. The DenseNet-121 comprises 4 dense blocks, which themselves comprise 6 to 24 dense layers.

• Dense block: A dense block comprises n dense layers. These dense layers are connected such that each dense layer receives feature maps from all preceding layers and passes it’s feature maps to all subsequent layers. The dimensions of the features (width, height) stay the same in a dense block.

• Dense layer: Each dense-layer consists of 2 convolutional operations.
  • 1 X 1 CONV (conventional conv operation for extracting features)
  • 3 X 3 CONV (bringing down the feature depth/channel count)

The CONV layer corresponds to the sequence BatchNorm->ReLU->Conv. A layer has each sequence repeated twice, the first with 1×1 Convolution bottleneck producing: grow rate x 4 feature maps, the second with 3×3 convolution. The authors found that the pre-activation mode (BN and ReLU before the Conv) was more efficient than the usual post-activation mode.

The growth rate (k= 32 for DenseNet-121) defines the number of output feature maps of a layer. Basically the layers output 32 feature maps which are added to a number of 32 feature maps from previous layers. While the depth increases continuously, each layer bring back the depth to 32.

• Transition layer: In between dense blocks, you find Transition layer. Instead of summing the residual like in ResNet, DenseNet concatenates all the feature maps. A transition layer is made of: Batch Normalization -> 1×1 Convolution -> Average pooling. Transition layers between two dense blocks ensure the down-sampling role (x and y dimensions halved), essential to CNN. Transition layers also compress the feature map and reduce the channels by half. This contributes to the compactness of the network.

Although Concatenating generates a lot of input channels, DenseNet’s convolution generates a low number of feature maps (The authors recommend 32 for optimal performance but world-class performance was achieved with only 12 output channels).

Key benefits:

• Compactness. DenseNet-201 with 20M parameters yields similar validation error as a 101-layer ResNet with 45M parameters.
• The learned features are non-redundant as they are all shared through a common knowledge.
• Easier to train because the gradient is flowing back more easily thanks to the short connections.

In this project, the model uses 320 x 320 X-Rays images and outputs predictions for each of the 14 pathologies as illustrated below on a sample image.

In order to run the model, I used an environment with tensorflow 1.15.0 and Keras 2.1.6. Model weights are provided in the repo.

I used a pre-trained model which performance can be evaluated using the ROC curve shown at the bottom. The best results are achieved for Cardiomegaly (0.9 AUC), Edema (0.86) and Mass (0.82). Ideally we want to be significantly closer to 1. You can check out below the performance from the ChexNeXt paper and their model as well as radiologists on this dataset.

Looking at unseen X-Rays, the model correctly predicts the predominant pathology, generating a somehow accurate diagnotic, highlighting the key region underlying its predictions. In addition to the main diagnostic (highest prediction), the model also predicts secondary issues similarly to what a radiologist would comment as part of his analysis. This can be either false positive from noise captured in the X-rays or cumulated pathologies.

The model correctly predicts Cardiomegaly and absence of mass or edema. The probability for mass is higher, and we can see that it may be influenced by the shapes in the middle of the chest cavity, as well as around the shoulder.

The model picks up the mass near the center of the chest cavity on the right. Edema has a high score for this image, though the ground truth doesn’t mention it.

Here the model correctly picks up the signs of edema near the bottom of the chest cavity. We can also notice that Cardiomegaly has a high score for this image, though the ground truth doesn’t include it. This visualization might be helpful for error analysis; for example, we can notice that the model is indeed looking at the expected area to make the prediction.

Performance from the ChexNeXt paper (model as well as radiologists):

Diagnose 14 pathologies on Chest X-Ray using Deep Learning. Perform diagnostic interpretation using GradCAM Method

This project is a complilation of several sub-projects from Coursera 3-course IA for Medical Specialization. The objective is to use a deep learning model to diagnose pathologies from Chest X-Rays.

The project uses a pretrained DenseNet-121 model able to diagnose 14 labels such as Cardiomegaly, Mass, Pneumothorax or Edema. In other words, this single model can provide binary classification predictions for each of the 14 labeled pathologies.

Weight normalization is performed to offset the low prevalence of the abnormalities among the dataset of X-Rays (class imbalance).

Finally the GradCAM technique is used to highlight and visualize where the model is looking, which area of interest is used to make the prediction. This is a tool which can be helpful for discovery of markers, error analysis, training and even in deployment.

The project uses chest x-ray images taken from the public ChestX-ray8 dataset. This dataset contains 108,948 frontal-view X-ray images of 32,717 unique patients. Each image in the data set contains multiple text-mined labels identifying 14 different pathological conditions. These in turn can be used by physicians to diagnose 8 different diseases. For the project we have been working with a ~1000 image subset of the images.

• 875 images to be used for training.
• 109 images to be used for validation.
• 420 images to be used for testing.

The dataset includes a CSV file that provides the ground truth labels for each X-ray.

DenseNet was introduced in 2017 in an award-winning paper by Gao Huang et al. 2018 called Densely Connected Convolutional Networks. The model was able to outperform previous architectures like ResNet (which I covered in a another project Skin Cancer AI dermatologist).

Regardless of the architectural designs of these networks, they all try to create channels for information to flow between the initial layers and the final layers. DenseNet, with the same objective, create paths between the layers of the network. Parts of this summary are can be found in this review.

• DenseNet key novelty: Densenet is a convolutional network where each layer is connected to all other layers that are deeper in the network
  • The first layer is connected to the 2nd, 3rd, 4th etc.
  • The second layer is connected to the 3rd, 4th, 5th etc.

Each layer in a dense block receives feature maps from all the preceding layers, and passes its output to all subsequent layers. Feature maps received from other layers are fused through concatenation, and not through summation (like in ResNets). Extracted feature maps are continuously added together with previous ones which avoids redundant and duplicate work.

This allows the network to re-use learned information and be more efficient. Such networks require fewer layers. State of the art results are achieved with as low as 12 channel feature maps. This also means the network has fewer parameters to learn and is therefore easier to train. Amongst all variants, DenseNet-121 is the standard one.

Key contributions of the DenseNet architecture:

• Alleviates vanishing gradient problem ( as networks get deeper, gradients aren’t back-propagated sufficiently to the initial layers of the network. The gradients keep getting smaller as they move backwards into the network and as a result, the initial layers lose their capacity to learn the basic low-level features)
• Stronger feature propagation
• Feature re-use
• Reduced parameter count

DenseNet is composed of Dense blocks. In those blocks, the layers are densely connected together: Each layer receive in input all previous layers output feature maps. The DenseNet-121 comprises 4 dense blocks, which themselves comprise 6 to 24 dense layers.

• Dense block: A dense block comprises n dense layers. These dense layers are connected such that each dense layer receives feature maps from all preceding layers and passes it’s feature maps to all subsequent layers. The dimensions of the features (width, height) stay the same in a dense block.

• Dense layer: Each dense-layer consists of 2 convolutional operations.
  • 1 X 1 CONV (conventional conv operation for extracting features)
  • 3 X 3 CONV (bringing down the feature depth/channel count)

The CONV layer corresponds to the sequence BatchNorm->ReLU->Conv. A layer has each sequence repeated twice, the first with 1×1 Convolution bottleneck producing: grow rate x 4 feature maps, the second with 3×3 convolution. The authors found that the pre-activation mode (BN and ReLU before the Conv) was more efficient than the usual post-activation mode.

The growth rate (k= 32 for DenseNet-121) defines the number of output feature maps of a layer. Basically the layers output 32 feature maps which are added to a number of 32 feature maps from previous layers. While the depth increases continuously, each layer bring back the depth to 32.

• Transition layer: In between dense blocks, you find Transition layer. Instead of summing the residual like in ResNet, DenseNet concatenates all the feature maps. A transition layer is made of: Batch Normalization -> 1×1 Convolution -> Average pooling. Transition layers between two dense blocks ensure the down-sampling role (x and y dimensions halved), essential to CNN. Transition layers also compress the feature map and reduce the channels by half. This contributes to the compactness of the network.

Although Concatenating generates a lot of input channels, DenseNet’s convolution generates a low number of feature maps (The authors recommend 32 for optimal performance but world-class performance was achieved with only 12 output channels).

Key benefits:

• Compactness. DenseNet-201 with 20M parameters yields similar validation error as a 101-layer ResNet with 45M parameters.
• The learned features are non-redundant as they are all shared through a common knowledge.
• Easier to train because the gradient is flowing back more easily thanks to the short connections.

In this project, the model uses 320 x 320 X-Rays images and outputs predictions for each of the 14 pathologies as illustrated below on a sample image.

In order to run the model, I used an environment with tensorflow 1.15.0 and Keras 2.1.6. Model weights are provided in the repo.

I used a pre-trained model which performance can be evaluated using the ROC curve shown at the bottom. The best results are achieved for Cardiomegaly (0.9 AUC), Edema (0.86) and Mass (0.82). Ideally we want to be significantly closer to 1. You can check out below the performance from the ChexNeXt paper and their model as well as radiologists on this dataset.

Looking at unseen X-Rays, the model correctly predicts the predominant pathology, generating a somehow accurate diagnotic, highlighting the key region underlying its predictions. In addition to the main diagnostic (highest prediction), the model also predicts secondary issues similarly to what a radiologist would comment as part of his analysis. This can be either false positive from noise captured in the X-rays or cumulated pathologies.

The model correctly predicts Cardiomegaly and absence of mass or edema. The probability for mass is higher, and we can see that it may be influenced by the shapes in the middle of the chest cavity, as well as around the shoulder.

The model picks up the mass near the center of the chest cavity on the right. Edema has a high score for this image, though the ground truth doesn’t mention it.

Here the model correctly picks up the signs of edema near the bottom of the chest cavity. We can also notice that Cardiomegaly has a high score for this image, though the ground truth doesn’t include it. This visualization might be helpful for error analysis; for example, we can notice that the model is indeed looking at the expected area to make the prediction.

Performance from the ChexNeXt paper (model as well as radiologists):

Vault has a concept called Database Secret Engine which supports widely used DBs. The database secrets engine is highly configurable & generates database credentials dynamically based on configured roles for a given TTL.

Things we’re concerned about:

• Deploying Vault on Kubernetes
• Microservice authentication to Vault
• Configuring authentication mechanism
• Configuring Database Secret Engine for Dynamic creds generation
• Consuming those generated creds into K8s workloads, our microservices
• Providing fine grained privileges to Microservices
• Periodically rotating credentials

In terms of available options on consuming Vault generated creds into microservices:

• Using Vault Agent Sidecar Injector

  • Sidecar Injection
  • Deployment change not required
  • Secrets rotation supported
  • Requires Helm

• Using Vault Agent Init Container (Preferable)

  • Uses Init Container
  • Deployment change required (only add some annotations)
  • Supports templating from secrets
  • Lightweight
  • Doesn’t require Helm
  • Manual/Automated intervention for Secret Rotation

• Using Secrets Store CSI Driver

  • Injects Daemonset to each K8s nodes
  • Deploy change required
  • Secrets rotation not supported
  • Secrets templating not supported
  • Requires Helm, CSI Driver, Vault CSI provider

• Re-writing Microservices to leverage on Vault APIs (Services must be Vault aware)

We’re going to deploying Vault with Raft HA Storage Backend with a self written auto unsealer & consuming DB creds using Vault agent init container. In production some Vault provider Auto Unseal mechanism should be used.

Note: The directory integration contains all the necessary manifest files for this demo, which is not part of the microservice itself.

We’ll create separate namespaces for the workloads:

NAME                 STATUS   AGE
app                  Active   5d2h 
db                   Active   5d3h 
monitoring           Active   3d3h
vault                Active   5d5h

# deploy vault on vault namespace

$ kubectl apply -f integration/vault/vault.yaml

serviceaccount/vault-agent-injector created
serviceaccount/vault created
configmap/vault-config created
clusterrole.rbac.authorization.k8s.io/vault-agent-injector-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/vault-agent-injector-binding created
clusterrolebinding.rbac.authorization.k8s.io/vault-server-binding created
role.rbac.authorization.k8s.io/vault-discovery-role created
rolebinding.rbac.authorization.k8s.io/vault-discovery-rolebinding created
service/vault-agent-injector-svc created
service/vault-active created
service/vault-standby created
service/vault-internal created
service/vault created
deployment.apps/vault-agent-injector created
statefulset.apps/vault created
poddisruptionbudget.policy/vault created
mutatingwebhookconfiguration.admissionregistration.k8s.io/vault-agent-injector-cfg created

# additionally we'll create the auto-unsealer to initialize & unseal the vault cluster
# note that: this is not required in case we use gcpckms auto unseal in gke

$ kubectl apply -f integration/vault/unsealer.yaml

Once the Vault resources are up & running:

$ kubectl get all -n vault

NAME                                       READY   STATUS    RESTARTS   AGE
pod/vault-0                                1/1     Running   0          3m6s
pod/vault-1                                1/1     Running   0          3m6s
pod/vault-2                                1/1     Running   0          3m6s
pod/vault-agent-injector-fbf44fc45-qcth7   1/1     Running   0          3m6s
pod/vault-unsealer-7b6998cdb5-6wz5g        1/1     Running   0          80s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   10.96.32.225    <none>        8200/TCP,8201/TCP   3m6s
service/vault-active               ClusterIP   10.96.116.55    <none>        8200/TCP,8201/TCP   3m6s
service/vault-agent-injector-svc   ClusterIP   10.96.160.167   <none>        443/TCP             3m6s
service/vault-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP   3m6s
service/vault-standby              ClusterIP   10.96.239.109   <none>        8200/TCP,8201/TCP   3m6s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           3m6s
deployment.apps/vault-unsealer         1/1     1            1           80s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-fbf44fc45   1         1         1       3m6s
replicaset.apps/vault-unsealer-7b6998cdb5        1         1         1       80s

NAME                     READY   AGE
statefulset.apps/vault   3/3     3m6s

# deploy DB on db namespace 

$ kubectl apply -f integration/postgres/postgres.yaml

service/postgres created
deployment.apps/postgres created

Once the postgres is up & running:

$ kubectl get all -n db

NAME                            READY   STATUS    RESTARTS   AGE
pod/postgres-56c58c445d-tx85b   1/1     Running   0          87s

NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/postgres   ClusterIP   10.96.54.28   <none>        5432/TCP   87s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/postgres   1/1     1            1           87s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/postgres-56c58c445d   1         1         1       87s

At this point, let’s create a table & write some dummy data in our DB (check the pg DB commands at the bottom for reference)

# enable kubernetes auth 

$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/


# configure kubernetes authentication

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Success! Data written to: auth/kubernetes/config

# Create Policy/Permission for the Microservice to use
# We'll provide fine grained permissions, only those required by our services

$ vault policy write app-policy integration/vault/app-policy.hcl

Success! Uploaded policy: app-policy

# Create a specifil Role for the Microservice 
# This will bind the service account name, namespace with the Policy created above

$ vault write auth/kubernetes/role/app-creds-reader-role \
    bound_service_account_names=app-auth \
    bound_service_account_namespaces=app \
    policies=app-policy \
    token_ttl=24h

Success! Data written to: auth/kubernetes/role/app-creds-reader-role

At this point, our app is ready to be authenticated to Vault. We’re going to create the service account mentioned here during app deployment.

# Enable the database secret engine 

$ vault secrets enable database

Success! Enabled the database secrets engine at: database/


# configure vault with plugin & connection information
# provided username, password must be privileged enough to create roles in DB
# We'll rotate this username, password instantly 

$ vault write database/config/postgres \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="db-reader-role" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.db.svc:5432/postgres?sslmode=disable" \
    username="postgres" \
    password="password"

Success! Data written to: database/config/postgres

# Create role that will be bounded to the generated username, password
# creation statements is DB specific valid statement based on the privileges microservice requires
# To be specific, we're providing only DB select permission to our microservice

$ vault write database/roles/db-reader-role \
    db_name="postgres" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="1h"

Success! Data written to: database/roles/db-reader-role

# Rotate the root password, we won't need this anymore
# You can verify by trying to login using the root creds provided to configure, it should fail.

$ vault write -force database/rotate-root/postgres

Success! Data written to: database/rotate-root/postgres

At this point, our DB Secret Engine is enabled & configured to generated dynamic creds on demand.

We’re going to use the Vault Agent Init container to inject creds into microservices. This requires only minimal changes in deployment, lightweight & works with K8s annotations intercepting MutatingWebhook configuration. Check out the deploy/app.yaml for more details.

# deploy the microservice on app namespace

$ kubectl apply -f deploy/app.yaml

serviceaccount/app-auth created
deployment.apps/app created

Once the app is up & running, we can check the app log to verify that it’s using the Vault generated creds.

2023/01/30 20:59:01 loading app
2023/01/30 20:59:01 loading db
2023/01/30 20:59:01 username:  v-kubernet-db-reade-zhOd64Scg1xFe2ORWQLj-1675112318
2023/01/30 20:59:01 password:  xREZ6QD28HaJL-x8P8Rb
2023/01/30 20:59:01 HTTP: Listening on port 8080

The service itself exposes the port 8080 & couple of endpoints to GET & UPDATE user. To verify that DB privilege binding is working, port-forward from the app:

kubectl port-forward -n app app-647984bb8-9dhgf 8080. Using Postman you can verify that, GET requests succeds while the POST doesn’t, since the privilege we binded allows our service only to GET from DB.

# creating a table called users

CREATE TABLE users
(
id          serial       not null unique,
title       varchar(255) not null,
name varchar(255)
);

# exect into postgres pod, connect using psql, authenticate with username, password

# inserting items into users table

INSERT INTO users(id, title, name)
VALUES (1, 'sre', 'sakib');

INSERT INTO users(id, title, name)
VALUES (2, 'manager', 'hanifa');

response should be in both case:

INSERT 0 1

# select all items from users table
select * from users;

postgres=# select * from users;
 id |  title  |  name  
----+---------+--------
  1 | sre     | sakib
  2 | manager | hanifa
(2 rows)

• https://developer.hashicorp.com/vault/docs/secrets/databases
• https://developer.hashicorp.com/vault/api-docs/secret/databases
• https://developer.hashicorp.com/vault/docs/platform/k8s/injector
• https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-sidecar
• https://developer.hashicorp.com/vault/docs/platform/k8s/injector/annotations
• https://www.hashicorp.com/blog/refresh-secrets-for-kubernetes-applications-with-vault-agent
• https://developer.hashicorp.com/vault/tutorials/raft/raft-storage
• https://developer.hashicorp.com/vault/tutorials/auto-unseal/autounseal-gcp-kms
• https://developer.hashicorp.com/vault/tutorials/operations/production-hardening

The Polish Radio Program 3 used to shape music taste of Polish listeners. Founded in 1982 and hosted by Marek Niedzwiecki, the Charts included songs that weren’t aired by any other radio station in Poland.

This app is meant to be a time machine for all listeners of Polish Radio Program 3, allowing them to listen to the chosen chart songs once again, on Spotify.

• crawling through a website with the charts of Polish Radio Program 3 and scraping the data of all of them
• writing web scraped data to json file
• reading json file and finding a chart aired on the date chosen by an user
• offering user a choice between the nearest dates, if the exact data hasn not been found but it is in the time frame of the charts airing
• finding Spotify uri-s of all the songs, using multiple queries options to guarantee the best search results
• creating and saving the playlist on an user’s account

• selenium webdriver module # to serve web crawler
• spotipy # to serve new playlists creation
• json # to serve json files
• datetime # to serve date search
• dotenv # to serve virtual environment variables
• pathlib # to serve files paths
• logging # to provide logs
• os # to serve system system files
• time # to serve proper web scraping, ensuring that the entire subpages are loaded

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.