Falling off a performance cliff with group_concat in Google Cloud SQL

I started to write this article as the performance of some of my mysql queries had gone off a cliff. This was related to my use of the GROUP_CONCAT function. This rolls up a set result set and shows the passed in row values concatenated. Handy for reporting, but not much else. In the highly memory constrained world of Google Cloud SQL using this on a well indexed set of tables (albeit it with multiple left outer joins in place) pushed a regularly executed query for a few hundred milliseconds to 6 to 9 seconds. Disaster.
I pulled the db over to my dev machine (copy of prod) and ran the query there. No problem. I am concluding that the GROUP_CONCAT was the bad guy here. I found another way to achieve my goal, but this cost me an afternoon and a lot of fretting before I got out of the woods.

MySql is not bad, but occasionally it bites.


Comments

Post a Comment

Popular posts from this blog

Building a choropleth map for Irish agricultural data

I wanted to create a choropleth map  using Irish data. The Irish Government has made a large quantity of data available at data.gov.ie. The set I chose was for Bovine TB across Ireland over a number of years. You can get geojson for a variety of countries and Ireland's county boundaries are available. All I had to do was get the data labels to match. In the data some counties are split into 2 parts. I merged and summed these data points to get the choropleth to work with normal county boundaries. You can see the results here . A link to the github is included at the top of the notebook.
Read more

Early Stopping with Keras

Steps to train a model There are a number of things you have to do when working on a Machine Learning problem. Import your data and transform it as appropriate Define  your model using something like Keras - this includes your chosen hyperparameter values Run the training phase Evaluate the model If at the end of this process the evaluation shows that your model is not sufficient for your needs you will need to make some changes and rerun the training.  Ideally you should have a way to spot that training is not going as you want and stop it. This is one of the uses of early stopping. The other is if your model has reached a sufficient quality standard for you to use and further improvements have slowed, stopped altogether or an unnecessary.  Early Stopping Keras provides a way of doing this using the EarlyStopping callback . There are a range of callbacks in Keras and you can define your own. Details are here . The EarlyStopping callback works by choosing a metric
Read more

AutoCompleteTextView backed with data from SQLite in Android

Image
This is a handy piece of code. As your database gets bigger you can allow the user to pick the correct value by typing. If the db gets very big, you can always increase the threshold, so that the query does not run until there will be a suitably sized resultset returned. Running it at a threshold of 1 is fine for very small resultsets.  Here is the code you need to back your AutoCompleteTextView with a cursor adapter. @Override protected void onResume() { super.onResume(); text = (AutoCompleteTextView) findViewById(R.id.autoCompleteTextView1); final AdapterHelper h = new AdapterHelper(this); Cursor c = h.getAllResults(); startManagingCursor(c); String[] from = new String[] { "val" }; int[] to = new int[] { android.R.id.text1 }; CursorAdapter adapter = new MyCursorAdapter(this, android.R.layout.simple_dropdown_item_1line, c, from, to); adapter.setFilterQueryProvider(new FilterQueryProvider() { public Cursor runQuery(Cha
Read more