Friday, March 17, 2017

Using Python to Automate Web Navigation

I use Selenium to log into and download files from websites.  Here's and example to get started.

Install Selenium Library


Assumptions

I'm assuming you have python installed and you have admin rights or are working inside a virtual environment (virtualenv) where you can install python libraries.

I also assume you can access the internet using the "pip" command.  If you're behind a proxy or firewall you might need to download Fiddler to tunnel your internet through the proxy.

Steps


  1. Open a command prompt
  2. Type: "pip install selenium"

Download the WebDriver for Your Browser

Assumptions

You're using Chrome.  Firefox may be able to skip this step as i don't think it needs a specific driver.

You can extract files from a zip file.  

Steps

  1. Go to the download site
  2. Download and extract the driver to a location you can access later with your program

Start Coding

Assumptions

I'm using the interactive python REPL that came with my installation of python called IDLE.  The code should work no matter what IDE or tool you're using.

I'll put code next to a description of each step.  For example:
1. Print hi to console: print("hi")

Steps

  1. Import the selenium web driver module: from selenium import webdriver
  2. Store a path to the web driver to a variable: chromedriver=r"C:\Test\chromedriver.exe"
  3. Instantiate the driver: driver = webdriver.Chrome( chromedriver )
  4. Open a web page: driver.get("http://stackoverflow.com")
That's it!  You can do a lot more with the selnium library, like log in to website, fill out forms, simulate button clicks, download files, etc.  

Wednesday, October 12, 2016

Sample Interview Questions


Round 1

  1. Look at Bloomberg ticker.  What can you tell me about this instrument
    1. IBM 10 P150 Equity
      1. It's a put option on ibm equity with 150 strike price
    2. When does it expire?
      1. 10/2016
    3. What day?
      1. third friday of the month
  2. Assume IBM stock is priced at 155.  Calculate the MV of 1000 standard contracts of the option above if it were a Call
    1. 1000 contracts * 100 shares * (155 price - 150 strike price) = 500,000
  3. Assume you have compliance rules that specify certain tickers you cannot trade.  These rules are specified at the firm level and at the fund level.  Firm level restrictions apply to all funds.  Write a query that shows all the funds and restricted tickers for each fund.
    1. select FundID, Ticker from Fund cross join Restrictions where FundID is null
    2. union 
    3. select FundID, Ticker from Restrictions where not FundID is null
  4. You are given a table of ticker, date, and daily P&L.  Find the top 10 tickers for each date by P&L.
    1. select * from (select date, ticker, rank = dense_rank() over (partition by date order by dailypnl desc) ) X where rank<=10
  5. You are given an array that represents a maze.  0 = space, 1 = wall, 2 = exit, 3 = start.  How would you write a program that solves the maze?
  6. In a hand of texas hold-em, you have two tens.  your opponent has queen, 9.  The flop shows J, 10, 8.  What is the probability of winning?


Round 2
  1. Design a database with the following data entities and constraints:
    1. DataEntities
      1. User - properties: user name
      2. Address - properties: street, city, state
      3. Order - properties: amount
      4. Office Location - properties: location name
    2. Constraints:
      1. User must have an address
      2. No two users can have the same address
      3. Every order has a user
      4. Not every user has an order
      5. User may have any number of office locations, including zero
    3. Solution:
      1. Users table has Primary key of User ID and a nullable address ID column with a Foreign Key and a unique constraint on address ID
      2. Address table has a primary key of address id
      3. Orders table has primary key of order id and non-nullable foreign key of user id.
      4. Office location table has primary key of office id
      5. UserOffice table has composite primary key of (userid, officeid) and foreign keys to user and office tables.
  2. Write a query and results based on tables in question 1 that would give name of all users, order id and amount of any orders for those users.  E.g. Username, Order ID, Amount
    1. Table Data:
      1. User
        1. ID = 1, Name = U1
        2. ID = 2, Name  = U2
        3. ID = 3, Name = U3
      2. Order
        1. ID = 101, UID = 1, Amount = 10
        2. 102, 2, 20
        3. 103, 1, 30
        4. 104, 2, 40
        5. 105, 1, 50
        6. 106, 2, 60
    2. Solution:
      1. Query
        1. select a.UName, b.OID, b.Amount
        2. from user a
        3. left join order b
        4. on a.UID=b.UID
        5. order by a. UNam, b.OID, b.Amount
      2. Results
        1. U1 , 101, 10
        2. U1, 103, 30
        3. U1 , 105, 50
        4. U2, 102, 20 
        5. U2 , 104,40
        6. U2 , 106,60
        7. U3, NULL, NULL
  3. Write a query and results based on tables and data in question 1 & 2 that would give name of all users and total amount of any orders for those users
    1. Solution:
      1. Query:
        1. select a. UName, total = Sum(Amount)
        2. from user a
        3. left join order b
        4. on a.UID=b.UID
        5. group by a. UName
      2. Results
        1. U1, 80
        2. U2, 120
        3. U3, NULL
  4. Same as question 3, except only show users where total is greater than 100
    1. Solution
      1. Query:
        1. select a. UName, total = Sum(Amount)
        2. from user a
        3. left join order b
        4. on a.UID=b.UID
        5. group by a. UName
        6. having sum(Amount)>100
      2. Results
        1. U2, 120
  5. Same as question 3, except there's a new user with a duplicate name: ID=4, Name=U1 and a new order: ID=107, UID=4, Amount = 100.  Ensure each unique user has their own total.
    1. Solution:
      1. Query:
        1. select a. UName, total = Sum(Amount)
        2. from user a
        3. left join order b
        4. on a.UID=b.UID
        5. group by a.UID, a. UName
      2. Results:
        1. U1, 80
        2. U2, 120
        3. U3, NULL
        4. U1, 100
  6. Write C# code that would give you the same output as question 3.  Assume you have C# objects with properties that are equivalent to columns in the tables.  Assume an object called DataService has methods getUsers() and getOrders() that return the user and order data, respectively.  
    1. Solution:
      1. see Main() below
  7. Same as 6 except assume the data set is now very large (e.g. # of users >100,000 and # orders > 1,000,000).  Hint: you will need a more efficient algorithm. 
    1. Solution:
      1. See Alternate() below

sample code for #6 and #7 

using System.Collections.Generic;

namespace MQuiz
{
  public class User
  {
  public int UID { get; set; }
  public string UName { get; set; }
  }
  public class Order
  {
  public int OID { get; set; }
  public int UID { get; set; }
  public float Amount { get; set; }
  }
  public class Program
  {
  public static void Main(string[] args)
  {
  var users = DataService.getUsers();
  var orders = DataService.getOrders();
  foreach (User u in users)
  {
  bool isMatched = false;
  foreach(Order o in orders)
  {
  if(u.UID == o.UID)
  {
  isMatched = true;
  System.Console.WriteLine(u.UName + " " + o.OID + " " + o.Amount);
  }
  }
  if(!isMatched) { System.Console.WriteLine(u.UName + " NULL NULL"); }
  }
  }
  public static void Alernative()
  {
  var users = DataService.getUsers();
  var orders = DataService.getOrders();

  var orderDictionary = new Dictionary<int, Dictionary<int, Order>>();
  foreach(Order o in orders)
  {

  Dictionary<int, Order> orderList;

  if (orderDictionary.ContainsKey(o.UID))
  {
  orderList = orderDictionary[o.UID];

  }
  else
  {
  orderList = new Dictionary<int, Order>();
  }

  orderList.Add(o.OID, o);

  orderDictionary.Add(o.UID, orderList);
  }

  foreach (User u in users)
  {
  if (orderDictionary.ContainsKey(u.UID))
  {
  var oDict = orderDictionary[u.UID];
  var o = oDict[u.UID];

  System.Console.WriteLine(u.UName + " " + o.OID + " " + o.Amount);
  }
  else
  {
  System.Console.WriteLine(u.UName + " NULL NULL");
  }

  }
  }
  }
  public static class DataService
  {

  public static List<User> getUsers() { return new List<User>(); }
  public static List<Order> getOrders() { return new List<Order>(); }
  }


}

Monday, August 29, 2016

The Best Tutorials and Demos for Building Web Apps

I'm creating this post for my friend who asked me where to find good resources on Web technologies.

It can be confusing and difficult to find information about technologies when you don't know what's out there.  Techies can be very opinionated / biased toward things they know best.  Here I will post a collection of the best tutorials from across the spectrum as I come across them.

Free

These tutorials are free and were collected from scattered sources.  

Knockout JS - This is a demonstration of knockout by its creator, Steve Sanderson, a Microsoft employee.  Knockout is a Javascript framework for data binding in a web application.  It provides functionality similar to Angular but in a different way that may be familiar to Microsoft developers.  Data binding and MVVM may be familiar to WPF or Silverlight developers.

Spotify Engineering Culture part 1 and part 2 - This is an in depth discussion of Agile programming for a larger team / product implemented by Spotify.  It's much more realistic than other agile descriptions because it's told by the team that actually implemented it.  It discusses how their Agile model changed from when they were small to the 1200+ group of engineers they are now.

Agile / Scrum Development - An illustration of Agile/Scrum methodology using a hypothetical example of a toy store.  It's a commercial for some project management tool, but a decent overview.

Pluralsight

These tutorials are hosted by PluralSight and may require a login.

Building Applications with ASP.NET MVC 4 - This tutorial by Scott Allen gives an overview of the Asp.Net MVC from installing visual studio to building sample apps.  I like Scott's tutorials in general because they have a fast pace to get up and running quickly while giving sufficient background to understand what's going on.  While this one in particular is using older technologies (MVC 4 has been superseded by 5 and 6) it is a good overview of the Asp.Net MVC framework that a lot of applications are built on.

Building a Web app with Asp.Net Core, MVC 6, and Angular - This tutorial gives a ground up explanation of how to use the latest technologies for building web apps.  The instructor gives a good overview early on about what Asp.Net core is all about and how it differs from older versions of Asp.Net.

Using Bootstrap CSS - Nice course explaining how boostrap works