Anup Saund

🇬🇧British Expat 🇨🇦Vancouver ❤ Gardening 💰Founder of gitSQL

Moving to Canada

I visited Vancouver 3 years ago on a business trip and fell in love with the City.

Sales Pitch

When I got back to the UK, I somehow convinced my partner that we should move to Canada! It wasn’t too much of a hard sell.

I then signed up to the express entry portal for Canada immigration and entered the skilled worker pathway. Check out their calculator to see if you qualify.

{ much paperwork later }

We fell short on points. Instead, we went down the Provincial Nominee Program (PNP) pathway, to support our application.

British Columbia, had a Tech Pilot where 32 categories of work were eligible for endorsement by the government. Their conditions were to have a valid job offer and pay $800 for the processing fee.

I got a job offer and shortly afterwards, we got an invitation to apply and received a COPR (Confirmation of Permanent Residency).

Bye bye England…

Flights booked, we got packing and moved to Vancouver. We have been here 2 months and it was hard going. Starting a new life with nothing but a few suitcases in an unfamiliar environment, sucks; but it was so worth it.

If you’re thinking of moving to a new country, for a better life. Just do it!

It won’t be an easy ride, but if you can stick it out to the end it will be so worth it. Anything is possible with enough effort, courage and grit. #believe

How to generate OpenAPI 3.0 (Swagger) spec from Vertx (Java) and serve it via Swagger UI

Java Micro Service API Generation

TL;DR - How to get java Vert.x to automatically generate a Open API v3 spec (a.k.a. Swagger) and serve it to Swagger UI, through Vert.x. Why I wanted to do it, and what benefits we saw.

Motivation

I joined a new role earlier this year as a Full Stack developer.

My colleagues had created RAML files for unit testing.

These files were also used to help document end points.

Unfortunately, the files were out of date;

  1. Developers found it a chore to keep the RAML files up to date.
  2. Testers would have to ask developers for example payloads for API requests.
  3. Automation engineers would ask developers for change details when automation failed.
  4. Nobody knew how many end points we had.
  5. Nobody wanted to write documentation for immature end points.

It was a real mess and something had to change!

I wanted to set up API documentation on the JAVA micro services in such a way that it would NOT be a burden on developers to keep up to date. I also wanted everyone a single source of truth.

Longer term, what I really wanted was to create SaaS ready API documentation…

Research and Development

I knew about swagger from previous projects; more importantly, Swagger UI.

My vision was to get Swagger UI into every micro service we had.

I searched the internet for ways to get a framework named Vert.x to generate Swagger.json. I found lots of JAVA related projects but only one specific to my requirements. It was good, but missing lots of implementation. The author had made a fantastic start to the project and wrote up omissions in the README.

Several weeks later…

vertx-auto-swagger was born.

A sample repository that would;

  1. Take JAVA annotations and generate OpenAPI 3.0 specification JSON.
  2. Serve the specifcation out on an end point.
  3. Serve Swagger UI out as a website.
  4. Use Swagger UI as a replacement to Postman.

Point 1 – I updated the implementation to include Schema definition and Examples of payloads.

Point 2 – I served the swagger specification out on /swagger.

Point 3 – I had a choice of either hosting a Swagger UI away from the code base or include Swagger UI within the codebase. I went with the latter as I did not want to battle with any CORS issues and wanted to reduce any DevOps overhead.

I took a distribution copy of Swagger UI and served it out through Vert.x as a static website.

Point 4 – I wanted to take advantage of the “Try it out” button in Swagger UI that would auto populate payloads with an example JSON. This was the end deliverable, as to give users a better alternative to using dekstop based applications such as Postman.

I wanted our swagger to have examples of request payloads so that a user could click and run examples. If the API changed then a developer could reflect the changes immediately for others to see.

The sample project uses no database calls, and mocks example requests.

The sample project uses Schema definitions with required field support.

This was a big must for us as I wanted to eventually have one routing method to validate all requests to a schema definition. The sample project has examples of this; if you try to send in an item that does not exist in the schema definition then a 400 is thrown to tell the user that a bad request was performed, with details on which item in the request was rejected.

Challenges

Path parameters!

Swagger UI has a “try it out” button. – It’s great, but it takes any Path parameters entered and substitutes them out using curly braces in the routing.

This is totally fine if one can rely on routing to use braces, but in my scenario, Vert.x insisted on using colon : to express parameters in the path.

I managed to get a fix in place for this, and sent a Pull Request in to Swagger API to get it adopted.

https://github.com/swagger-api/swagger-js/pull/1338

Unfortunately it was rejected as the Swagger spec does not support colon’s.

Hi @anupsaund!

We’re unable to add support for this, since it’s not part of the Swagger/OpenAPI Specification.

If you’d like support for colon notation in parameters, the first step is to start a discussion on the OpenAPI Specification repository: https://github.com/OAI/OpenAPI-Specification/issues/new.

If it’s added to a future version of the specification, we can then add support for it here 😄

Also note that you could (and for now, should) modify the output of your generator to translate colon path parameters to the templating format that OpenAPI calls for today.


I’m going to close this out since it’s not possible to accept such a change at this time. Thanks for your efforts!

I then took to hacking the code into my minified version of Swagger UI. It’s not ideal but at the moment, it works. BTW, it was too hard to switch colons to braces in Vert.x.

I also created a ticket over at OpenAPI for colon support to be considered for inclusion into the specification. https://github.com/OAI/OpenAPI-Specification/issues/1681

Did it actually work for us?

It took a few months of development work but we now have a fully documented customer facing API with JWT authentication that is being used cross functionally.

We also went ahead and moved the generator code into a common library so that it can be used across micro services.

We all suspected that the API’s we had written would require attention in areas with respects to correct response codes and 500 errors. We now have full visibility of what needs doing and surprisingly, the mountain of tech debt we suspected turned out to be a few mole hills of work! We now have visibility of what we need to incorporated into out sprint planning, in order to make our API’s robust and eventually customer facing.

Summary & Repo Link

I hope this article inspires others to create a similar holistic solution to API documentation and web API contracts. It was very well received by my colleagues and senior management as a MUST feature for us to incorporate into everything we do.

The repository link is below. Please check it out, there is a more comprehensive read me in there from a technical perspective.

Github Project: https://github.com/anupsaund/vertx-auto-swagger

Mobile App Server

Serve Google Android and Apple iOS applications to Mobile Devices.

https://www.npmjs.com/package/mobile-app-server

The Problem

One day at the office, I noticed my QA colleague struggle to get an Apple iOS app onto their iPhone for testing.

Typically we use online services such as https://www.diawi.com/ to present us with a QR code.

We then use this to download an .ipa application onto the mobile device using a QR code reader.

Earlier that day I was generating Google Android and Apple iOS packages via a Jenkins build server.

That’s when I thought;

“Wouldn’t it be great to serve the files out locally for download?”

The solution

I was looking for an on-premise Diawi service!

I searched high and low to see if anyone had done something like this before. Nothing came close to what I needed but I did find a useful package that enabled reading of IPA and APK packages.

I used this package and set up a Expressjs  application that would serve out files on a web interface.

Serving an Apple iOS package required a secure server so I set up the application over HTTPS as default with instructions on how to set up a self certified key and certificate.

Main Features

  1. Automatically discovers new Apps on page refresh, simply add files to path.
  2. Designed for Local Intranet usage, but could be deployed to cloud.
  3. Provides package information and QR Code per app.
  4. Secured on HTTPS – Requires an SSL certificate (Self Cert or Full).

Summary

If you are looking for an in house web based file server to serve mobile applications then this could be a valid solution. The service watches a folder to check for changes, new files, deletion of files. Subsequently browsing to the website will reflect the contents of the folder being watch.

Roadmap

  1. Drag and Drop app addition of files via Web Interface.
  2. App Icons.
  3. Extended package information.

Would you like any more features?

Get in touch, I would be happy to develop this further.

Alternatively, feel free to fork/pull request enhancements into the repository.

https://github.com/anupsaund/mobile-app-server

gitSQL – Source Control for SQL Server

TL;DR: How to export SQL Server objects and data to flat file for version control. A blog post behind the gitSQL tool, why it came about and the journey so far.

 

Problem – I need to source control my database

Earlier this year I was thinking about how to automate the extraction of SQL objects and data from SQL Server into flat file. I wanted to do this so I could save the files to GIT and then track changes.

I also wanted a bit more than that too – I wanted to be able to build a database from source control.

I don’t ask for much right?

Research

I searched and searched and found the following products on the market.

 

Liquibase is free – but I couldn’t manage to configure it to my requirements. If i did manage then this blog post wouldn’t exist.

I tried each of the other products and they all felt good – but only if you are willing to spend between $200 – $1000.

I didn’t want to spend any money, or at least, I could not justify spending any money right now.

I then set on it to distill the very essence of what these apps were offering and what is going on underneath the hood.

What did I do next?

You probably guessed it –

I created my own App, and launched it in April this year  as

http://www.gitsql.net

My main reason for creating the app was because I wanted something simple for myself and I thought it would be great to share it;

gitSQL has 3 screens and a command line interface coming in the next release.

The CLI will help with continuous automation, and general streamlining of source control.

gitSQL is Free for up to 20 tables, and 10 other objects, with an UNLIMITED version @ $40.

At first, I didn’t want to charge anything for it but I soon realised that I would need to support the app with potential bug fixes and future features.

Therefore – the UNLIMITED licence is simply to cover development costs and hosting costs.

Any happy customers?

One of my early customers had a fantastic experience with the product, so much so that he kindly blogged about his experience.

Read all about it:

https://rcbsql.wordpress.com/2015/04/24/version-control-for-this-dummy/

I have also had a lot of direct messages on Facebook and Twitter asking about current features and CLI options for the future – This is really rewarding for me because it means that gitSQL is working for others like I thought it might do. #success

I also had a request for adding Sequences as an object type – through gitHub, and after a bit of testing i worked out it could be added and it’s made it into the latest release. #github issue 4

What’s next?

I would really like to include a PostgreSQL, MySQL and maybe an Oracle version(s) of gitSQL to make a more comprehensive suite of source control apps.

I am currently working on the PostgreSQL version which should be released Q2 2016. The MySQL version may also be released at the same time if I can engineer things correctly. The end result may be one app for them all – or several individual apps bundled into one single download. #not_decided_yet

The future products will follow the same model of being fully featured for FREE, with a small licence cost for UNLIMITED usage (again just to cover my development/hosting/support costs).

I hope gitSQL works for you – and if it doesn’t please let me know over at gitHub, Facebook or Twitter.

Sick of coding? … Try growing veg!

So what happened?

Year before last – i found myself looking at an overgrown plot at my local allotments.

My immediate thought was – wow, this would be hard work if i took it on and it would probably take me away from my coding projects… but i went ahead an rented the plot anyway.

I’m now in year 2 of having the plot and I’ve also had a baby boy, my sister’s wedding and a house move in between.

Surprisingly, I’ve managed to get more throughput on my coding projects that I had previously?

Doesn’t make sense right?

I love spending time at the allotments, it’s hard work, i build up a sweat and the results take a long time to come, but there is a clear process to success.

  1. Clear the ground
  2. Sow seeds (greenhouse or ground)
  3. Remove weeds
  4. Add nutrients
  5. Crop

What does this have to do with coding?

1. Clear the ground

I find myself clearing up coding environments, setting up GIT, a database, so on and so forth before I start a new project. Some aspects are automated but essentially what I am doing is clearing the ground.

2. Sow seeds

Most of my code visualization is done away from the computer, when I’m in the shower, driving around, washing up or at the allotments. I find this to be the key to a successful coding session at the computer. Visualization is the key word here, because when I plant seeds at the allotments, i may not see any sprout for a week or so – but i have the plant visualized in my head. I liken this to writing classes in code which have functions that do not do anything yet but they are there to help visualize the end product.

3. Remove weeds

Software needs weeding time to time, by which I mean, re-factoring and debugging. Sometimes – early on, i find myself re-writing code just so I can engineer it in such a way that it can be used for other projects. Sometimes I don’t if i know it’s not going to be used again, but in any case, I have to approach the coding session as if I’m going to remove weeds.

4. Add nutrients

In the gardening context, soil needs nutrients and sun and water. These are the basic requirements.

In the coding context, the human requires nutrients in order to produce quality code. We are not machines, and it’s important to remember that. It’s also good to know our optimal time in the day when we are at our coding best.

In the code base context, nutrients usually come in the form of inspiration from other projects, integration with other code bases and collaboration with other coders.

5. Crop

Cropping in the software sense is knowing when a product is ready for release. It’s as simple as that.
We have to go through some test phases and make sure the product does what it’s suppose to do – i liken it to testing a tomato firmness, or peeling a bean pod to see the contents.

How did I get more throughput on my coding?

I made my sessions count!

Baby duties, wedding planning, moving house, going to work, going to gym, spending time with family and friends… that’s all important to me, so I had to make sure that when I allocated time for coding, it really had to count, because i knew i may not get that window again for several days.

Make it count!

 

 

Website Design Trends 2010 / 2011?

Website trends have stabilised over the years thanks to companies such google, wikipedia, amazon and facebook to name a few.

Web developers, designers and users are generally more aware of user interface design – what works and what doesn’t?

Here are a few examples of websites which I have found that seem to follow a ‘trend’.

BBC

BBC

BBC Website

ebuyer

ebuyer

ebuyer Website

xfactor

xfactor

Xfactor Website

Sunday Times

The Sunday Times

The Sunday Times Website

Royal Mail

royalmail

Royal Mail Website

Tesco Direct

tescodirect

Tesco Direct Website

BBC iPlayer

iplayer

BBC iPlayer Website

What do they have in common?

1) Oversized animated header

2) Large typography for website headings

3) Column grid design for the entire website (as used by newspaper editors such as The Sunday Times)

4) Oversized footer section which contains a sitemap

5) Search bar located at the top right

It seems to me that a lot of websites are adopting the oversized footer section of the website as their navigation area. BBC and ITV both have oversized footer sections which contain links through to internal pages. The links are displayed in columns which can be read quickly by the reader (Pacing). Is it easier for the visitor to see links on the footer?

Wikipedia recently completed a usability study which showed that the search bar located at the top right was the most intuitive place that a user would look for when searching a website. (See their explanation here). We can see that the BBC and Sunday Times websites have search bars at the top right of their websites. Does this mean that we should be putting search bars at the top right?

Oversized animated headers seem popular too – as the content of the header changes with a transiation. In a way – this shows the visitor more content during their visit without them having to click anywhere. This has got to be a plus? no? Especially if a visitor only stays for a few seconds. Are oversized, animated headers better than static headings?

Typography on websites have become extremely popular over recent years- especially with designers. I imagine this is because website developers now have the tools available to place designer typography onto websites. Text replacement seems common on some websites as a means of adopting dynamic typography driven by CMS. Is this better practice than using web fonts?

Familiarity

… do I need to say any more?

Most of my google analytics statistics show that the average website visitor will leave a website on the homepage – without ever clicking through onto other pages.

Hmm – maybe I need to make better websites? *gulp*

Saying that – I imagine this is a general issue with all websites with respects to converting visitors. There’s an old analogy I like to use.

We normally have to head out to the grocery store to get bread and milk. Supermarkets place their milk and bread at the back of the store so we have to go past everything else first. If we’re not on our guard we reach the checkout with several items along with the milk.

A good website will “Give them Milk”… – a reason to come back.

Perhaps we need to be giving users familiar design (like the supermarkets do)

What do you think?

Stripslashes in SQL for MYSQL and ORACLE

Problem

Today, I wanted to strip slashes from a comment field using SQL (INSTEAD of using php stripslashes).

Why you ask?

I was working on an application which had not considered stripslashes for formatting output.

So I had a look and realised. Do I change 40,000 lines of code to add Stripslashes, or 1 line at the SQL select statement?

OK, slight exaggeration, but you get what I mean.

Solution

MySQL

Select
Replace(field_name, '\', '') as stripped_field
from
table;

Oracle

Select
Replace(field_name, '', '') as stripped_field
from
table;

PHP EXTRA

If using in PHP add an extra backslash like this;

// strip slashes as part of the SQL select statement
$sql = "Select Replace(field_name, '\', '') as stripped_field from table";

Please get in touch if you know of an easier way to do this, for example, Oracle Functions, or MySQL functions.

Many thanks and happy coding.

I remember the days before ADODB

I remember a time when SQL was written for a database.

Oracle SQL was different to Access SQL

mySQL was different to SQL Server SQL

… then somebody said no, stop, enough is enough.

Sometime in 2009 ADODB was born for PHP. The 1 wrapper for ALL database types. Ah, just what we wanted.

Wait a minute? This is ADO, didn’t microsoft do this in 1996? The short answer is yes. Database abstracting is old in theory, tried out by microsoft, worked to a degree but never fully made it in the world (for whatever reason).

This brings us up to ADODB  – the promise of a database wrapper that will make it through, and iron out the issues that microsofts offering had.

Today I got my hands dirty with ADODB and I liked it. Looks and feels just like any other wrapper such as ADO.

Actually, I think it’s the same same. So so… così così.

One thing for sure. It makes hot swapping between Oracle/MySQL (which is what i’m doing right now) a breeze.

That’s why i’m writing about it.

I like it… how about you?

PRE tags, the quick way to keep formatting

I get asked quite often if there is a way to show CODE on a website, with it’s formatting in place?

The short answer is <pre>

It can output free text (i.e. non server side code) in a pre-formatted way. Yep, PRE = Pre-formatted.

On top of that, Preformatted also renders the php print_r() output in a nicely nested unordered list, which is great for debug purposes;

e.g.

&nbsp; echo "
<pre>";&nbsp;&nbsp; &nbsp;print_r($arr);&nbsp; echo "</pre>
"; 

I am sure there are other uses for the <pre> tag too, I have just scratched the surface.

Give it a go – let me know if you find other uses for it.

Thanks.

BCC – Blind Carbon Copy – The forgotten field!

The BCC field can usually be found underneath the CC field.

Sound’s obvious right?

You already know about the field, yes?

Great!

— but not everyone knows about it, or how to use it. I say this because I have had emails sent to me where ALL the contacts have been in the To: field.

This is bad practice because;

1) It violates the data protection act (we didn’t give permission for our email address to be shared)

2) Looks unprofessional

3) Opens us up to spam!

Whenever I get an email like this, it is usually because the person sending it is inexperienced/hasn’t had the training in proper use of email. So for those who use the BCC field correctly well done. For everyone else, please learn.

For MASS email, please enter ALL your to contacts into the BCC field, and then add your own address to the TO field. This way, ALL emails will appear to be sent to yourself, with blind carbon copies going off to everyone else.

Happy emailing.

Page 1 of 2

Powered by WordPress & Theme by Anders Norén