MongoDB World 2017: Building Micro-Services Based ERP System

Jerry M Reghumadh of Capiot did a talk on building micro-services from the ground up.  The legacy system that his group replaced was monolithic and rigid.  The solution the Capiot team proposed to the client placed each component of the ERP system into its own atomic component.  Everything on the platform that was built was an API.  These API's were very "chatty".

The engineering decisions that were made included the choice of NodeJS and MongoDB as the base technologies for this platform.  NodeJS was selected in part, because of its small footprint.  This lowered the barrier to entry for the application.  Java was considered, but it was too heavy for the needs of the project.  MongoDB was selected for the data persistence layer because it saves data as documents and it did not require the marshaling and unmarshaling of data.  MongoDB also allowed the implementation team to use a flexible schema.  MongoDB offered greater ease of clustering and sharding versus other available options for this project.  This allowed the developers to implement this without relying on a dedicated database administrator.
The technology stack included:

• NodeJS
• ExpressJS
• Swagger.io
• Mongoose.js
• Passport.js

The team implemented a governance model that forced any exposed API to be exposed in Swagger.  This prevented the proliferation of "rogue" API's.  Any API not exposed in swagger would not work properly in the system.  Mongoose allowed the team to enforce a schema.

Ronan Bohan from MongoDB and Vaidy Krishnan from Tableau presented the Jumpstart presentation.  The BI connector was just shipped by MongoDB.  The mission of Tableau is to harness the power of data.  There are three core tenets of the development philosophy are:

1. Connectivity to access all data
2. Design software for deeper thinking
3. Ability to scale data and provide analytics on that data

There is a focus on Big Data.  The goal is to provide focus and make the data discoverable.  Vaidy stated that Tableau is about data access.  Tableau will transfer the data from the application layer to the business actionable level.  Tableau has been adopted by 55,000 clients all over the world.

Ronan began a discussion of how to connect MongoDB data to the Tableau platform.  Tableau was originally designed to work with structured data from relational databases.  MongoDB is designed to store semi-structured data.  This made MongoDB and Tableau incompatible.  This problem has been solved by the BI connector.  This allows MongoDB's semi-structured data to connect to the Tableau visualization platform.

Ronan provided the audience with a demo using the BI Connector to link the MongoDB data to the Tableau visualization tool.  He displayed a DRDL Definition file which is a YAML  file that is used to map the MongoDB data to a schema that applications that consume structured data can understand.

Detailed information on the MongoDB BI Connector can found on the MongoDB website. (https://www.mongodb.com/products/bi-connector)

MongoDB World 2017 Chicago Keynote

The keynote was given by Megan and Richard in the Hyatt Grand Ballroom.  Megan stated that there will be six tutorials.  MongoDB World is paperless this year, so a mobile app will be used to find sessions, tutorials and give feedback.

Tom Schenk is the first keynote speaker.  (CTO City of Chicago).  WindyGrid is a system built for the city of Chicago that allows city staff to analyze and visualize data piped into the system from 17 different input applications.  MongoDB drives this application.  Data that is sent to the system includes weather, 911 calls, 311 calls, etc.  Predictive analytics are used to drive the operations of the city.  Data is captured and managed in MongoDB. The code for the predictive analytics for food safety has been released as open source code.  (https://github.com/Chicago/opengrid)

Tom transitioned by introducing the CEO of MongoDB, Dev Ittycheria.  Dev started by going over the history of the internet and technology from the introduction of Netscape in 1995.  Around 2000, fiber optic communications were built out and made it cost effective for technology work to be distributed around the world.  2007 was a watershed year because the cost of computing power and storage dropped to a point where many new types of business concepts became feasible.  MongoDB was founded in 2007.

2007 document model, 2010 distributed framework, 2012 aggregation framework, 2013 Management and security, 2014 WiredTiger, 2015 Doc validation, 2016 MongoDB Atlas.

Dev talked about the growth of MongoDB over the last few years and how widespread the platform has become.  Last year there was a 30% increase in people taking MongoDB University classes.
Shawn Melamed (Morgan Stanley).  Shawn stated that Morgan Stanley started using MongoDB 5 years ago.  One of the attractions of the database is the flexibility of how the data can be structured.

2017 -  Elliot  Horowitz announcing new products and services coming out of MongoDB

• Business Intelligence - (Note: Tableau 3.0 came out that allows this visualization tool that allows to easily connect to MongoDB)
• MongoDB Charts has been announced which allows charts to be built in a document-centric view.  This allows developers and analysts to quickly build dashboards based on MongoDB.  The tool has an intuitive drag and drop interface and allows users to build common types of charts (bar charts, pie charts)  This feature will be part of MongoDB 3.6 and will be released later this year.
• In MongoDB 3.6, the $lookup operator has been enhanced to allow more advanced and sophisticated queries.  Enhancements have been added to updating arrays
• Fully expressive array updates
• Schema ( schemaless, semi-structured, unstructured ).  MongoDB 3.6 will support JSON Schema.
• Retryable writes - this guarantees that a write will happen at least one time and only one time if desired.
• In 3.6, it will only be possible to connect to localhost by default.  This should close some current security holes in the product.
• Change streams in action
• MongoDB Stitch (NEW Service) - Backend as a service
  • REST API MongodDB
  • Configuration based authentication, privacy, and security
  • Service composition
  • Anywhere you want
  • Available on MongoDB Atlas today in beta

MongoDB 3.6 is scheduled to ship in, November 2017.

• MongoDB Atlas is the recommended way of running MongoDB in the cloud.
  • Free Tier
  • Performance Tier
  • Queriable backups
  • Supported on AWS
  • Support added for Microsft Azure and Google Cloud
  • Hosted BI Connector
  • LDAP authentication  allows the connection with on-premise enterprise systems
  • Cross-regions support (coming next year)
  • Cross-cloud support (coming next year)

Cailin Nelson, Vice President, Cloud Engineering spoke and announced support for Microsoft Azure and Google Cloud in addition to AWS.  Atlas has been optimized for each cloud platform.  The new performance advisor allows administrators and developers to track down performance issues easily.
What has changed since 2007

• The Web as a 1st class citizen
• Mobile
• IoT

Polymorphic data can appear in the same collection withing MongoDB, which is pretty powerful.
The hashtag #MDBW17 will be used to post announcements.