So, you have embarked into your Deep Learning journey and perhaps you are navigating through the concepts of Gradient Descent, Back-propagation and so forth. After all the theory you are eager to get your environment ready to do some actual ‘deep learning hard work’ and you have no idea where to start. You are in the right place then. This short tutorial has been put together for Mac user (sorry Windows aficionados) and will provide you with what you need to get started.

Yes, you need Python!

Sure you know that Python is the key programming language when it comes to Machine and Deep Learning. Make sure you have our beloved HomeBrew:

/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install Python 3 (with this version, pip3 will be automatically installed)

brew install python3

Virtual Environment

In order to keep things clean and contain all your deep learning related dependencies in one space, it is useful to use virtual environments.

pip3 install virtualenv virtualenvwrapper

You will also need to modify your bash profile file:

vim ~/.bash_profile

by adding the following:

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Next step is to create a virtual environment for your deep learning project:

mkvirtualenv cv -p python3

This will create a virtual environment named cv and in order to come out of such instance, you will need to type the command deactivate

Some Additional Dependencies

You will also need to install cmake to be able to use dlib, a C++ toolkit containing Machine Learning algorithm:

brew install cmake

Additionally, you will need to download X11 to display the image’s outputs from both dlib and opencv target=”_blank”

Let’s install the real stuff

Situate yourself inside your virtual environment by typing the following:

workon cv

Some additional dependencies should be taken care of:

pip install numpy h5py pillow scikit-image

Finally, we can install OpenCV:

pip install opencv-python


Then, we will be installing Dlib, Tensorflow and Keras:

pip install dlib
pip install tensorflow
pip install keras

Keras, in particular, is a user friendly, beginner library for Machine Learning and Deep Learning models that runs on top of Tensorflow. Happy Machine Learning modelling 🙂

Perhaps the most basic question for beginners when learning about Machine Learning and Deep Learning.

https://towardsdatascience.com/clearing-the-confusion-ai-vs-machine-learning-vs-deep-learning-differences-fce69b21d5eb

SparkSQL is a Spark module for working with structure data and it can also be used to read columnar data format such as Parquet files.  Here a number of useful commands that can be run from the spark-shell:

#Set the context

val sqlContext = new org.apache.spark.sql.SQLContext(sc)

#Read the parquet file in HDFS and

val df = sqlContext.read.parquet(“hdfs://user/myfolder/part-r-00033.gz.parquet”).printSchema

#Show the top 10 rows of data from the parquet file

df.show(10, false)

#Convert to JSON and print out the content of 1 record

df.toJSON.take(1).foreach(println)

Source: Jenkins Best Practices – Practical Continuous Deployment in the Real World — GoDaddy Open Source HQ

Why Avro For Kafka Data?

I have been recently asked whether it is possible to use break (and continue as well) in a loop with Scala and it occurred to me that I have never come across such a case. Coming from Java, I do know how to employ break and continue in a while loop, for example, so why would it be different in Scala, considering that it is builds on top of the JVM? It is actually a bit more complicated than that. Although Scala does not specifically have the keywords break and continue, it does offer similar functionality through scala.util.control.Breaks.

Here is an example of how to use break from the Class Breaks, as follows:

import scala.util.control.Breaks._
import java.io._

val in = new BufferReader(new InputStreamReader(System.in))

breakable {
  while (true) {
    println ("? ")
    if (input.readLine() == "") break
  }
}

In Java, the above would corresponding to this:

BufferedReader in =
   new BufferedReader(new InputStreamReader(System.in));
   while (true) {
     if (in.readLine() == "") break
   }

The breakable function has become available from Scala 2.8 onwards and before that we would have tacked the issues mostly through 2 approaches:

• by adding a boolean variable indicating whether the loops keeps being valid;
• by re-writing the loop as a  recursive function;

Happy Scala programming 🙂

As a Mac user, I usually run Apache NiFi using one of the two approaches:

• by standing up a Docker container;
• by downloading and installing locally on your Mac;

Running a NiFi Container

You can install Docker on Mac via Homebrew:

brew install docker

Alternatively it is possible to download the Docker Community Edition (CE): an easy to install desktop app for building, packaging and testing dockerised apps, which includes tools such as Docker command line, Docker compose and Docker Notary

After installing Docker, this will let you pull the NiFi image:

docker pull apache/nifi:1.5.0

Next, we can start the image and watch it run:

docker run -p 8080:8080 apache/nifi:1.2.0

Downloading and Installing NiFi locally

Installing Apache NiFi on Mac is quite straightforward, as follows:

brew install nifi

This assumes that you have Homebrew installed. If that is not the case, this is the command you will need:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null

Here is where NiFi has been installed:

/usr/local/Cellar/nifi/your-version-of-nifi

Some basic operations can be done with these commands:

bin/nifi.sh run, it runs in the foreground,

bin/nifi.sh start, it runs in the background

bin/nifi.sh status, it checks the status

bin/nifi.sh stop, it stops NiFi

Next step, whatever approach you took at the beginning, is to verify that your NiFi installation/dockerised version is running. This is as simple as visiting the following URL:

localhost:8080/nifi

Happy Nif-ing 🙂

Machine Learning’s ‘Amazing’ Ability to Predict Chaos
http://go.squidapp.co/n/eSFPx8E

Download SQUID – Your News Buddy
squidapp.co/getSQUID

A while ago I was tasked to write a small application in order to connect to an Oracle Database and perform a set of simple queries. For such a task, I have employed the DAO (Data Access Object) pattern and a corresponding DAO Interface. A basic Java client, in turn, calls the instantiation of such DAO class, which implements a the DAO interface. As follow, the application in its internal details:

Oracle DB Client 

[code language=”java”]
package oracledb.connection.client;
import oracledb.connection.dao.OracleDB_DAO;

public class OracleConnectionClient {

public static void main(String[] args) throws Exception {

OracleDB_DAO dao = new OracleDB_DAO();
dao.readPropertiesFile();
dao.openConnection();
dao.getDBCurrentTime();
dao.getFirstNameAndLastNameFromCustomers();
dao.closeConnection();

}
}
[/code]
The Data Access Object (DAO) implementation. The method[code]readPropertiesFile()[/code]

parses a properties file containing the access credentials and DB connection details.

[code language=”java”]
package oracledb.connection.dao;

import java.io.*;
import java.sql.*;
import java.util.Properties;

public class OracleDB_DAO implements OracleDB_DAO_Interface {

public static String SAMPLE_SELECT_QUERY = “SELECT * FROM CUSTOMERS WHERE FirstName = ‘Eliott’ AND LastName = ‘Brown'”;

private static String driverClass = “oracle.jdbc.driver.OracleDriver”;
private Connection connection;
private static String dbUrl;
private static String userName;
private static String password;

static String resourceName = “dbconnection.properties”;

/**
* Read the properties Initialise the DAO
*
* @throws IOException
* @throws ClassNotFoundException
*/

public void readPropertiesFile() throws IOException, ClassNotFoundException {

ClassLoader loader = Thread.currentThread().getContextClassLoader();
Properties props = new Properties();
InputStream resourceStream = loader.getResourceAsStream(resourceName);
{
props.load(resourceStream);
}

// Return the properties
dbUrl = props.getProperty(“dburl”);
userName = props.getProperty(“dbuser”);
password = props.getProperty(“dbpassword”);

// Load the
Class.forName(driverClass);
}

/*
* (non-Javadoc)
*
* @see oracledb.connection.dao.OracleDB_DAO_Interface1#openConnection()
*/
@Override
public void openConnection() throws SQLException {

// get the connection to the database
System.out.println(“Establishing the Connection to the Database”);
try {
connection = DriverManager.getConnection(dbUrl, userName, password);
System.out.println(connection);
} catch (SQLException ex) {
ex.printStackTrace();
}
}

/*
* (non-Javadoc)
*
* @see oracledb.connection.dao.OracleDB_DAO_Interface1#closeConnection()
*/
@Override
public void closeConnection() throws SQLException {
if (connection != null) {
// close the connection
connection.close();
}
}

/*
* (non-Javadoc)
*
* @see oracledb.connection.dao.OracleDB_DAO_Interface1#
* getFirstNameAndLastNameFromCustomers()
*/
@Override
@SuppressWarnings(“resource”)
public ResultSet getFirstNameAndLastNameFromCustomers() throws SQLException, IOException {
// create the prepared stmt
Statement stmt = connection.createStatement();
// assign the query to a variable
String sql = SAMPLE_SELECT_QUERY;
// execute the query
ResultSet rs = stmt.executeQuery(sql);
System.out.println(“This print the ResultSet for getPlanByMSISD ” + rs);
@SuppressWarnings(“unused”)
PrintWriter csvWriter = new PrintWriter(new File(“sample.csv”));

stmt.close(); // close statement
return rs;
}

/*
* (non-Javadoc)
*
* @see oracledb.connection.dao.OracleDB_DAO_Interface1#getDBCurrentTime()
*/
@Override
public String getDBCurrentTime() throws SQLException, IOException {
String dateTime = null;
// create the prepared stmt
Statement stmt = connection.createStatement();
ResultSet rst = stmt.executeQuery(“select SYSDATE from dual”);
while (rst.next()) {
dateTime = rst.getString(1);
}
// close the resultset
System.out.println(“This prints the dateTime from the DB ” + dateTime);
rst.close();
return dateTime;

}
}
[/code]
The DAO Interface that defines the standard operations to be performed on a model object:

[code language=”java”]
package oracledb.connection.dao;

import java.io.IOException;
import java.sql.ResultSet;
import java.sql.SQLException;

public interface OracleDB_DAO_Interface {

/**
* Open the Dao Connection
*
* @param
* @throws SQLException
* @throws IOException
*/
void openConnection() throws SQLException;

/**
* Close the connection
*
* @throws SQLException
*/
void closeConnection() throws SQLException;

/**
* Get the resultset from the the select query
*
* @throws SQLException
* @throws IOException
*/
ResultSet getFirstNameAndLastNameFromCustomers() throws SQLException, IOException;

/**
* Get the Current Time via DB Query
*
* @return
* @throws SQLException
* @throws IOException
*/
String getDBCurrentTime() throws SQLException, IOException;

}
[/code]