Visible improvements in this build:

 - Significantly reduce the queries that have to be performed, improving lag - especially over slow connections (Issue #118; see new controller info under headline code changes).
 - Fix Issue #117 properly (export numeric quoting - we now have access to column types and so can quote appropriately).
 - Fix Issue #145 (loss of unsigned/null/default attributes when reordering columns).
 - Fixes Issue #90 (support for filtering DECIMAL column types)
 - Improve table scrolling speed when the table contains long items. (Added a NSFormatter to automatically truncate strings > 150 chars for display purposes only)
 - Improved SQL compatibility - for example /* C style comments */ are now correctly ignored in imports and custom queries.
 - Add text and symbols emphasising that the table info pane / status view row count is an approximation (partially addresses Issue #141)
 - Fixes a major memory leak whenever opening or scrolling tables containing text/blob data.
 - SQL import is now faster (SQL parsing part is 3x faster).
 - Speed up SQL export (1.5x faster for numeric data; 1.1x faster for string data) and slightly speed up CSV export (~1.1x faster).
 - Display sizes on the status view using the byte size formatter, as per table info pane.
 

Headline code changes:
 - Add a new NSMutableString subclass, SPSQLParser.  See the header file for documentation and overview, but in short it's a centralised place for SQL parsing.  Centralises and improves parsing, improves comment support, improves quoting support.  Despite the improved featureset this is also faster than the previous distributed implementations - for example, when used to replace the old splitQueries:, > 3x speedup.
 - Implement a new controller which handles a structure and status cache for the current table, and provides structure parsing for specified tables.  This cache is now used throughout the code, reducing the queries that have to be performed and providing additional information about the table structure for use; I think it also improves column type format slightly.
 - The table info pane and the status view now draw all their data from the cache.


Tweaks:
 - Table encoding is now detected directly instead of being derived from the collation - increased accuracy and cope with the DEFAULT encoding.
 - Comments and formatting cleaned up in bits I was working on, obviously.
 - A couple of methods - particularly [tablesListInstance table] and [tableDocument encoding] - have been renamed to avoid conflicts and fix code warnings.


Future improvements now possible:
 - As we now have access to column types and other information, we can provide per-type behaviour where desired.
 - The table parsing doesn't currently pull out comments or table indices, together with one or two other attributes.  Some of this would be useful for display; some, such as indices, could be used to draw the table structure view as long as we're happy discarding a couple of columns (ie cardinality!)

1 files changed, 118 insertions, 20 deletions

diff --git a/Interfaces/English.lproj/DBView.xib b/Interfaces/English.lproj/DBView.xib
index 392632b3..30e81caa 100644
--- a/Interfaces/English.lproj/DBView.xib
+++ b/Interfaces/English.lproj/DBView.xib
@@ -9612,6 +9612,9 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 			<object class="NSCustomObject" id="622219357">
 				<string key="NSClassName">SPTableInfo</string>
 			</object>
+			<object class="NSCustomObject" id="1006422530">
+				<string key="NSClassName">SPTableData</string>
+			</object>
 		</object>
 		<object class="IBObjectContainer" key="IBDocument.Objects">
 			<object class="NSMutableArray" key="connectionRecords">
@@ -10586,14 +10589,6 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 				</object>
 				<object class="IBConnectionRecord">
 					<object class="IBOutletConnection" key="connection">
-						<string key="label">tableSourceInstance</string>
-						<reference key="source" ref="392169872"/>
-						<reference key="destination" ref="103234030"/>
-					</object>
-					<int key="connectionID">855</int>
-				</object>
-				<object class="IBConnectionRecord">
-					<object class="IBOutletConnection" key="connection">
 						<string key="label">queryProgressBar</string>
 						<reference key="source" ref="427689665"/>
 						<reference key="destination" ref="440898411"/>
@@ -11066,14 +11061,6 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 				</object>
 				<object class="IBConnectionRecord">
 					<object class="IBOutletConnection" key="connection">
-						<string key="label">tableDumpInstance</string>
-						<reference key="source" ref="601471102"/>
-						<reference key="destination" ref="225526897"/>
-					</object>
-					<int key="connectionID">1239</int>
-				</object>
-				<object class="IBConnectionRecord">
-					<object class="IBOutletConnection" key="connection">
 						<string key="label">tableTabView</string>
 						<reference key="source" ref="427689665"/>
 						<reference key="destination" ref="714795046"/>
@@ -11828,6 +11815,78 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 					</object>
 					<int key="connectionID">4701</int>
 				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDocumentInstance</string>
+						<reference key="source" ref="1006422530"/>
+						<reference key="destination" ref="427689665"/>
+					</object>
+					<int key="connectionID">4705</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableListInstance</string>
+						<reference key="source" ref="1006422530"/>
+						<reference key="destination" ref="438574515"/>
+					</object>
+					<int key="connectionID">4706</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="438574515"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4707</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="427689665"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4708</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="103234030"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4709</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="622219357"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4710</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="936957743"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4711</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="392169872"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4712</int>
+				</object>
+				<object class="IBConnectionRecord">
+					<object class="IBOutletConnection" key="connection">
+						<string key="label">tableDataInstance</string>
+						<reference key="source" ref="225526897"/>
+						<reference key="destination" ref="1006422530"/>
+					</object>
+					<int key="connectionID">4713</int>
+				</object>
 			</object>
 			<object class="IBMutableOrderedSet" key="objectRecords">
 				<object class="NSArray" key="orderedObjects">
@@ -16884,6 +16943,12 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 						<reference key="object" ref="143557124"/>
 						<reference key="parent" ref="632761638"/>
 					</object>
+					<object class="IBObjectRecord">
+						<int key="objectID">4702</int>
+						<reference key="object" ref="1006422530"/>
+						<reference key="parent" ref="1043842561"/>
+						<string key="objectName">SPTableData</string>
+					</object>
 				</object>
 			</object>
 			<object class="NSMutableDictionary" key="flattenedProperties">
@@ -17874,6 +17939,7 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 					<string>47.IBPluginDependency</string>
 					<string>47.ImportedFromIB2</string>
 					<string>4700.IBPluginDependency</string>
+					<string>4702.IBPluginDependency</string>
 					<string>495.IBAttributePlaceholdersKey</string>
 					<string>495.IBPluginDependency</string>
 					<string>495.ImportedFromIB2</string>
@@ -19381,6 +19447,7 @@ IGRvIHlvdSB3YW50IHRvIGFkZCBmb3IgdGhpcyBmaWVsZD8</string>
 					<string>com.apple.InterfaceBuilder.CocoaPlugin</string>
 					<reference ref="9"/>
 					<string>com.apple.InterfaceBuilder.CocoaPlugin</string>
+					<string>com.apple.InterfaceBuilder.CocoaPlugin</string>
 					<object class="NSMutableDictionary">
 						<string key="NS.key.0">ToolTip</string>
 						<object class="IBToolTipAttribute" key="NS.object.0">
@@ -19805,7 +19872,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 				</object>
 			</object>
 			<nil key="sourceID"/>
-			<int key="maxID">4701</int>
+			<int key="maxID">4713</int>
 		</object>
 		<object class="IBClassDescriber" key="IBDocument.Classes">
 			<object class="NSMutableArray" key="referencedPartialClassDescriptions">
@@ -19901,7 +19968,6 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>queryFavoritesSheet</string>
 							<string>queryFavoritesView</string>
 							<string>queryHistoryButton</string>
-							<string>tableDumpInstance</string>
 							<string>tableWindow</string>
 							<string>textView</string>
 							<string>valueSheet</string>
@@ -19920,7 +19986,6 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
-							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">
@@ -19988,6 +20053,27 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 					</object>
 				</object>
 				<object class="IBPartialClassDescription">
+					<string key="className">SPTableData</string>
+					<string key="superclassName">NSObject</string>
+					<object class="NSMutableDictionary" key="outlets">
+						<bool key="EncodedWithXMLCoder">YES</bool>
+						<object class="NSMutableArray" key="dict.sortedKeys">
+							<bool key="EncodedWithXMLCoder">YES</bool>
+							<string>tableDocumentInstance</string>
+							<string>tableListInstance</string>
+						</object>
+						<object class="NSMutableArray" key="dict.values">
+							<bool key="EncodedWithXMLCoder">YES</bool>
+							<string>id</string>
+							<string>id</string>
+						</object>
+					</object>
+					<object class="IBClassDescriptionSource" key="sourceIdentifier">
+						<string key="majorKey">IBProjectSource</string>
+						<string key="minorKey">Source/SPTableData.h</string>
+					</object>
+				</object>
+				<object class="IBPartialClassDescription">
 					<string key="className">SPTableInfo</string>
 					<string key="superclassName">NSObject</string>
 					<object class="NSMutableDictionary" key="outlets">
@@ -19995,6 +20081,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 						<object class="NSMutableArray" key="dict.sortedKeys">
 							<bool key="EncodedWithXMLCoder">YES</bool>
 							<string>infoTable</string>
+							<string>tableDataInstance</string>
 							<string>tableDocumentInstance</string>
 							<string>tableList</string>
 							<string>tableListInstance</string>
@@ -20005,6 +20092,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">
@@ -20075,8 +20163,8 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>multipleLineEditingButton</string>
 							<string>removeButton</string>
 							<string>tableContentView</string>
+							<string>tableDataInstance</string>
 							<string>tableDocumentInstance</string>
-							<string>tableSourceInstance</string>
 							<string>tableWindow</string>
 							<string>tablesListInstance</string>
 						</object>
@@ -20227,6 +20315,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>syntaxView</string>
 							<string>syntaxViewContent</string>
 							<string>tableContentInstance</string>
+							<string>tableDataInstance</string>
 							<string>tableDumpInstance</string>
 							<string>tableSourceInstance</string>
 							<string>tableStatusInstance</string>
@@ -20267,6 +20356,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 							<string>NSTabView</string>
 							<string>id</string>
 							<string>id</string>
@@ -20391,6 +20481,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>singleProgressSheet</string>
 							<string>singleProgressText</string>
 							<string>tableContentInstance</string>
+							<string>tableDataInstance</string>
 							<string>tableDocumentInstance</string>
 							<string>tableListView</string>
 							<string>tableSourceInstance</string>
@@ -20446,6 +20537,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">
@@ -20512,6 +20604,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>removeFieldButton</string>
 							<string>removeIndexButton</string>
 							<string>structureGrabber</string>
+							<string>tableDataInstance</string>
 							<string>tableSourceView</string>
 							<string>tableTypeButton</string>
 							<string>tableWindow</string>
@@ -20536,6 +20629,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">
@@ -20572,6 +20666,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>sizeIndex</string>
 							<string>sizeMaxData</string>
 							<string>tableCreatedAt</string>
+							<string>tableDataInstance</string>
 							<string>tableName</string>
 							<string>tableType</string>
 							<string>tableUpdatedAt</string>
@@ -20591,6 +20686,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">
@@ -20638,6 +20734,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>customQueryInstance</string>
 							<string>tabView</string>
 							<string>tableContentInstance</string>
+							<string>tableDataInstance</string>
 							<string>tableDocumentInstance</string>
 							<string>tableDumpInstance</string>
 							<string>tableSourceInstance</string>
@@ -20659,6 +20756,7 @@ Y2hhbmdlIHRoZSBvcmRlcg</string>
 							<string>id</string>
 							<string>id</string>
 							<string>id</string>
+							<string>id</string>
 						</object>
 					</object>
 					<object class="IBClassDescriptionSource" key="sourceIdentifier">