We were hired to convert a large UK accountancy firm’s many interconnected Excel spreadsheets into a robust SQL Server database project. Part of the project included exporting data from Excel to SQL Server.
The intricacies of doing this were immense.
The London accountancy firm
The accountancy firm is a large, respected company based in London. They manage the accounting of thousands of independent contractors across the UK, doing everything from tax returns to day-to-day payments for them.
They deal with massive amounts of data.
Using Excel as a database was no longer an option
Understandably, the company had outgrown its in-house Excel spreadsheets and systems.
We had already created a fairly sophisticated Excel database solution for them which manipulated and processed the incoming and outgoing data efficiently. But the company had reached a level of expansion where there was simply too much data for Excel to deal with speedily and rapidly.
And there were other concerns as well, such as:
- Portability of the data, should the company wish to move to a different platform in the future.
- Advanced reporting which was becoming ever more difficult, given the sheer quantity of data they were dealing with.
- Ease of backups.
- Maintaining the integrity of the data as the company grew and more staff gained access to raw figures.
- Security of data.
Excel had been doing a great job until the numbers hit several million records per spreadsheet.
We had already created a database in Excel. Now we needed to take that “database” and put it into an actual database — something like MySQL or SQL Server or Oracle.
The decision to switch from Excel to SQL Server
Excel tends to “play better” with Microsoft products, so we were not keen to use MySQL.
Eppert Consulting had previously completed a gargantuan forecasting project for a German company whose preferred database was Oracle. There are several intricacies involved in connecting Excel to Oracle, more so than with SQL Server.
The accountancy firm was comfortable with Microsoft products, and so it was decided that we would connect Excel to SQL Server.
That’s when the work began.
If I were to detail for you every step of the project’s life cycle, this article would end up being a book. The entire process was both fascinating and rewarding.
Excel is a powerful tool, but combined with SQL Server as its back-end data provider, it is without a doubt the most effective tool any company can have in its arsenal. It can do everything from creating quotations to running sophisticated forecasting software which increases sales.
The project, looking back at it, feels now almost legendary in its scope, even though it did not feel this way at each step of the process.
Here are some details of what we did.
The initial data import from Excel to SQL Server
Although both Excel and SQL server have built-in tools for importing and exporting data, I would not recommend using these for a project with so much data.
The task is best done with VBA programming.
Once the data has been imported, it well behooves the programmer to spend as much time as is necessary verifying the correctness of that imported data.
This step cannot be undervalued. There is so much potential for minor importing errors which could have snowball effects later on. You will save yourself much debugging time if you verify the data fully and completely at this stage. A fresh set of eyes helps. Excel formulas and some direct VBA programming checks are also useful.
Daily importing and exporting from Excel to SQL Server and vice versa
Once the data has been imported into Excel, it needs to be pulled back into Excel in order to work with it.
Nothing beats Excel’s myriad functions for working with data. The accountancy firm was keen to move the data to SQL Server, but dropping Excel as an accountancy tool was never in question.
There are several ways to regularly export and import from SQL to Excel, and vice versa. The most common one (although we do not recommend it) is to open up a connection to the SQL Server using VBA code, pull the data, close the connection, and keep doing this for each transaction.
This is inefficient.
Increasing Excel’s speed was imperative
Speed was a key factor in deciding to move from Excel to SQL server, and constantly opening and closing connections would slow the software down due to the enormous amount of transactions this company was performing a minute.
We recommend pulling the data from SQL Server into a hidden workbook in Excel using VBA programming. Do this when the program first opens up, then periodically refresh the data to maintain consistency.
There are fine details on how this can be done without data conflicts, but that is beyond the scope of this article.
The key thing is to “flatten” the data. Excel is worked with most easily when all the data is on a single sheet. SQL Server is the exact opposite. By the very definition of a relational database, best practices in SQL Server dictate that one must separate the data into tables, and relate them one to the other. (The technical term for this is “normalising” the data.)
These two paradigms are in direct conflict with each other.
Interfacing between Excel and SQL Server
If you wish to use Excel with SQL Server as a back-end, you will need to provide an interface between the two systems which “flattens” the data so that it is mostly in one sheet in Excel. To send data to SQL Server, you need a reverse system which “normalises” it into separate tables on the way back.
This might seem like quite some programming, and it can be. But once the initial elements are in place, the code can be re-used for all data transfers between Excel and SQL Server.
Excel is the boss in an accountancy firm. It is unlikely that you’ll end up using even a tenth of SQL Server’s many features. When connecting Excel to SQL Server, the programming task is made easier in the long run by doing the majority of work in Excel’s VBA programming language. This makes future debugging easier.
Further Excel integrations
But it didn’t stop there. By the time we had completed the bulk of the Excel to SQL Server integration, the tool had reached a stage of such value that the company requested further modifications to it.
Each of the modifications was small, and took very little time, comparatively, to implement. But looking back at the completed product, I am still amazed at how much we achieved. I am also amazed at how unbelievably flexible Microsoft Excel can be when it comes to connecting and integrating it with other tools and software such as Sage, Adobe Acrobat, Amazon Web Services, etc.
Here are just a handful of the many integrations Eppert Consulting programmed into this mammoth and invaluable piece of Excel accounting software.
Integrating Excel with Outlook
One of the first requests was to integrate Excel with Outlook — preparing a client’s tax return, for example, and then sending the return to them in a PDF format using Outlook.
We did this in the Visual Basic for Applications programming language, of course. Advanced tasks such as these can only be done using Excel’s programming features.
We ended up programming many integrations between Excel and Outlook. Here are some of the features that were required for each or all of those integrations:
- Correct formatting of the prepared Excel document so it looked professional when exported to PDF.
- Automatically emailing it to an email address associated with the client (stored in the underlying SQL Server database).
- Various email templates which could be edited by the client.
- Email templates automatically selected when clicking the “Send” button, based off options filled in, in the Excel spreadsheet.
- Bulk emailing of attached financial figures (either in Excel or in PDF format) utilising an underlying data store.
- A sophisticated document approval process, resulting in a completed document finally being sent out by email once it had been verified and checked at various points.
Connecting Excel to bulk-emailing and SMS software
The company uses a bulk emailing service, independent of Microsoft Outlook. It also uses a bulk SMS service which requires messages to be uploaded to a third-party company’s website in a specific format.
We programmed a feature to create these messages with one click, formatting the upload-file within Excel itself, using VBA programming code.
Importing banking data into Excel — integration with HSBC Connect, RBS and other financial institutions
One of the most ambitious tasks we had was to import financial transactions from HSBC, Royal Bank of Scotland and other financial institutions into Excel.
Excel also had to export payment data directly to these financial institutions.
Each institution had its specific way of accepting data, as well its own security practices. We programmed several features so that Excel could both receive and send data for thousands of bank accounts, for which this accounting company had a mandate.
Programming Excel to work with Amazon Web Services
The company has produced several hundred-thousand documents over the years. Amazon Web Services was chosen as a secure location to store these documents.
The Excel tool needed a function where it could directly upload generated documents to Amazon Web Services’ Simple Storage servers.
To achieve this, we had to program an external tool in Microsoft’s .Net programming language, because Amazon did not provide an interface for Excel.
Well, we did it, and we also synchronised the data for the uploaded documents with our local SQL Server database so that we always had a local record with which to compare Amazon’s data to ours.
We also maintain separate backups of all documents. The Excel software takes care of placing each copy of the document — backup and original — where it belongs.
Connecting Excel to PHP, MySQL and Microsoft .Net websites
Meanwhile, web technology continued to progress at a staggering speed. The accountancy firm’s clients requested more and more that their documents be available to download online.
To do this, it was necessary to create a second online database to store information regarding each user’s documents. We had to program functions to:
- Notify users of their generated documents.
- Ensure that only authorised users could download documents.
- Upload documents into a secure location where they could be downloaded by authenticated users.
We also programmed a feature where clients could approve or disapprove financial transactions using an online platform.
We used a MySQL database to store data because the firm already had a PHP website in place.
This eventually led to a Microsoft .Net website, an additional SQL database, on and on and on…
The project kept growing, each time adding value to the firm, and improving its ROI.
Excel can connect to anything — anything at all!
There was never any dearth of ideas, and the software kept growing. Each new feature requested was a challenge, and I am happy to say that Eppert Consulting rose to meet those challenges.
No matter the request, we found a way to achieve it. This speaks more for Excel’s versatility than to our programming skills. Although an in-depth knowledge of Excel and all its potentials is absolutely necessary to come up with solutions to unusual problems — to think “outside the box.”