refactor!: Refactor field count handling in CsvReader

Replaced `ignoreDifferentFieldCount` with `allowExtraFields` and `allowMissingFields` for improved control over record validation. Changed the default to NOT allow this.

Author: osiegmar

docs/src/content/docs/architecture/interpretation.md

@@ -89,15 +89,17 @@ Consider the following CSV snippet as an illustration of varying field counts:
 ```
 header_a,header_bCRLF
 value_a_1CRLF
-value_a_2,value_b_2CRLF
+value_a_2,value_b_2,value_c_2CRLF
 ```
 
 In this example, `value_a_1` likely belongs to `header_a`, and `header_b` does not have a value for the first data
-record. However, this is just an assumption.
+record. However, this is just an assumption. Field `value_c_2` does not even have a corresponding header.
 
-By default, FastCSV handles scenarios with different field counts by ignoring them (see
-`CsvReaderBuilder.ignoreDifferentFieldCount(boolean)`). This is done to accommodate this frequently occurring case. To
-ensure no misinterpretation, you can disable this behavior, causing an exception to be thrown when reading such data.
+To ensure no misinterpretation, FastCSV does not allow extra or missing fields in a record by default.
+This means that the above example would result in a `CsvParseException` when reading it with FastCSV.
+
+However, this behavior can be changed by setting `CsvReaderBuilder.allowExtraFields(boolean)`
+and `CsvReaderBuilder.allowMissingFields(boolean)` to `true`.
 
 ### Empty lines
 

docs/src/content/docs/guides/Examples/reading-ambiguous-data.mdx

@@ -10,8 +10,8 @@ There are a few widespread ambiguities in CSV files:
    See the [Empty lines section of the CSV Interpretation](/architecture/interpretation/#empty-lines) page for more information.
 2. **Empty fields**: CSV files can contain empty fields, which are fields that contain no data.
    See the [Empty fields / null values section of the CSV Interpretation](/architecture/interpretation/#empty-fields--null-values) page for more information.
-3. **Missing fields**: CSV files can contain missing fields, which are fields that are not present in a record.
-   See the [Different field count of the CSV Interpretation](/architecture/interpretation/#different-field-count) page for more information.
+3. **Extra or missing fields**: CSV files can contain a different number of fields in each record.
+   See the [Extra fields section of the CSV Interpretation](/architecture/interpretation/#extra-fields) page for more information.
 
 FastCSV is very aware of these ambiguities and provides ways to handle them.
 

docs/src/content/docs/guides/Examples/skip-non-csv-head.mdx

@@ -21,9 +21,10 @@ Strictly speaking, such a file **is not a valid CSV file** as defined by the CSV
 
 The main problem with those files is:
 
-- An exception would be thrown unless the options `ignoreDifferentFieldCount()` and `skipEmptyLines()` are set.
 - When working with named fields, the very first line (`This is an example of a CSV file that contains`)
   would be interpreted as the actual header line.
+- An exception would be thrown unless the options `allowExtraFields(true)` is set, as some lines have
+  more fields than the first line.
 
 FastCSV comes with two features to handle such files:
 

docs/src/content/docs/guides/basic.md

@@ -51,7 +51,8 @@ CsvReader.builder()
     .commentStrategy(CommentStrategy.SKIP)
     .commentCharacter('#')
     .skipEmptyLines(true)
-    .ignoreDifferentFieldCount(false)
+    .allowExtraFields(false)
+    .allowMissingFields(false)
     .allowExtraCharsAfterClosingQuote(false)
     .detectBomHeader(false)
     .maxBufferSize(16777216);

docs/src/content/docs/guides/upgrading.md

@@ -13,6 +13,17 @@ For a full list of changes, including new features, see the [changelog](https://
 - The minimum Java version has been raised from 11 to 17
 - This also raised the required Android API level from version 33 (Android 13) to 34 (Android 14)
 
+## Ignoring different field counts
+
+FastCSV 4.x no longer ignores different field counts by default, ensuring that data is not misinterpreted.
+
+You can change this behavior by calling `allowExtraFields(true)` and `allowMissingFields(true)` in the `CsvReaderBuilder`.
+These methods provide more control over how to handle different field counts in CSV data than the previous (now removed) `ignoreDifferentFieldCount()` method.
+
+:::caution
+As the default has changed, you may need to check your code and your desired behavior.
+:::
+
 ## Internal buffer flushing
 
 In FastCSV 2.x and 3.x, the CsvWriter instantiated via `CsvWriterBuilder.build(Writer)` flushed the internal buffer to the `Writer` after each record.

example/src/main/java/example/ExampleCsvReaderWithFaultyData.java

@@ -8,34 +8,27 @@ public class ExampleCsvReaderWithFaultyData {
 
     private static final String DATA = """
         foo,bar
-        only one field followed by some empty lines
-
-
-        bar,foo
+        foo
+        foo,bar,baz
         """;
 
     public static void main(final String[] args) {
-        System.out.println("Reading data with lenient (default) settings:");
-        CsvReader.builder()
-            .ofCsvRecord(DATA)
-            .forEach(System.out::println);
-
-        System.out.println("Reading data while not skipping empty lines:");
-        CsvReader.builder()
-            .skipEmptyLines(false)
-            .ofCsvRecord(DATA)
-            .forEach(System.out::println);
-
-        System.out.println("Reading data while not ignoring different field counts:");
+        System.out.println("Reading data with default settings:");
         try {
             CsvReader.builder()
-                .ignoreDifferentFieldCount(false)
                 .ofCsvRecord(DATA)
                 .forEach(System.out::println);
         } catch (final CsvParseException e) {
-            System.out.println(e.getMessage());
-            System.out.println(e.getCause().getMessage());
+            System.out.println("Exception expected due to different field counts:");
+            e.printStackTrace(System.out);
         }
+
+        System.out.println("Reading data while not ignoring different field counts:");
+        CsvReader.builder()
+            .allowExtraFields(true)
+            .allowMissingFields(true)
+            .ofCsvRecord(DATA)
+            .forEach(System.out::println);
     }
 
 }

lib/src/intTest/java/blackbox/reader/AbstractCsvReaderTest.java

@@ -80,36 +80,38 @@ void immutableResponse() {
             .isInstanceOf(UnsupportedOperationException.class);
     }
 
-    // different field count
+    // allow extra fields
 
-    @ParameterizedTest
-    @ValueSource(strings = {
-        "foo\nbar",
-        "foo\nbar\n",
-        "foo,bar\nfaz,baz",
-        "foo,bar\nfaz,baz\n",
-        "foo,bar\n,baz",
-        ",bar\nfaz,baz"
-    })
-    void differentFieldCountSuccess(final String s) {
-        assertThatNoException().isThrownBy(() -> readAll(s));
+    @Test
+    void allowNoExtraFields() {
+        assertThatThrownBy(() -> readAll("foo\nfoo,bar"))
+            .isInstanceOf(CsvParseException.class)
+            .hasMessage("Exception when reading record that started in line 2")
+            .hasRootCauseInstanceOf(CsvParseException.class)
+            .hasRootCauseMessage("Record 2 has 2 fields, but first record had 1 fields");
     }
 
     @Test
-    void differentFieldCountSuccess2() {
-        crb.ignoreDifferentFieldCount(false);
-        assertThatNoException().isThrownBy(() -> readAll("foo\nbar"));
+    void allowExtraFields() {
+        crb.allowExtraFields(true);
+        assertThatNoException().isThrownBy(() -> readAll("foo\nfoo,bar"));
     }
 
-    @Test
-    void differentFieldCountFail() {
-        crb.ignoreDifferentFieldCount(false);
+    // allow missing fields
 
-        assertThatThrownBy(() -> readAll("foo\nbar,\"baz\nbax\""))
+    @Test
+    void allowNoMissingFields() {
+        assertThatThrownBy(() -> readAll("foo,bar\nfoo"))
             .isInstanceOf(CsvParseException.class)
             .hasMessage("Exception when reading record that started in line 2")
             .hasRootCauseInstanceOf(CsvParseException.class)
-            .hasRootCauseMessage("Record 2 has 2 fields, but first record had 1 fields");
+            .hasRootCauseMessage("Record 2 has 1 fields, but first record had 2 fields");
+    }
+
+    @Test
+    void allowMissingFields() {
+        crb.allowMissingFields(true);
+        assertThatNoException().isThrownBy(() -> readAll("foo,bar\nfoo"));
     }
 
     // allow extra characters after closing quotes

lib/src/intTest/java/blackbox/reader/AbstractSkipLinesTest.java

@@ -54,15 +54,15 @@ void multipleRecordsNoSkipEmpty() {
     @ParameterizedTest
     @ValueSource(strings = {",\nfoo\n", ",,\nfoo\n", "''\nfoo\n", "' '\nfoo\n"})
     void notEmpty(final String input) {
-        crb.quoteCharacter('\'');
+        crb.allowMissingFields(true).quoteCharacter('\'');
         final CsvRecordHandler cbh = CsvRecordHandler.of(c -> c.fieldModifier(FieldModifiers.TRIM));
         assertThat(crb.build(cbh, input).stream()).hasSize(2);
     }
 
     @ParameterizedTest
     @ValueSource(strings = {",\nfoo\n", ",,\nfoo\n", "''\nfoo\n", "' '\nfoo\n"})
     void notEmptyCustomCallback(final String input) {
-        crb.quoteCharacter('\'');
+        crb.allowMissingFields(true).quoteCharacter('\'');
         final AbstractBaseCsvCallbackHandler<String[]> cbh = new AbstractBaseCsvCallbackHandler<>() {
             private final List<String> fields = new ArrayList<>();
 

lib/src/intTest/java/blackbox/reader/CsvReaderBuilderTest.java

@@ -79,7 +79,7 @@ void builderToString() {
             .isEqualTo("""
                 CsvReaderBuilder[fieldSeparator=,, quoteCharacter=", \
                 commentStrategy=NONE, commentCharacter=#, skipEmptyLines=true, \
-                ignoreDifferentFieldCount=true, allowExtraCharsAfterClosingQuote=false, \
+                allowExtraFields=false, allowMissingFields=false, allowExtraCharsAfterClosingQuote=false, \
                 trimWhitespacesAroundQuotes=false, detectBomHeader=false, maxBufferSize=16777216]""");
     }
 
@@ -128,7 +128,8 @@ void chained() {
             .commentStrategy(CommentStrategy.NONE)
             .commentCharacter('#')
             .skipEmptyLines(true)
-            .ignoreDifferentFieldCount(false)
+            .allowExtraFields(false)
+            .allowMissingFields(false)
             .allowExtraCharsAfterClosingQuote(false)
             .ofCsvRecord("foo");
 

lib/src/intTest/java/blackbox/reader/RelaxedCsvReaderTest.java

@@ -35,7 +35,7 @@ void readerToString() {
         assertThat(crb.ofCsvRecord(""))
             .asString()
             .isEqualTo("CsvReader[commentStrategy=NONE, skipEmptyLines=true, "
-                + "ignoreDifferentFieldCount=true, parser=RelaxedCsvParser]");
+                + "allowExtraFields=false, allowMissingFields=false, parser=RelaxedCsvParser]");
     }
 
 }