Skip to content

Commit 75f4633

Browse files
kazuki43zookazuki43zoo
committed
Support adding at any position
1 parent 4d3312d commit 75f4633

File tree

17 files changed

+166
-48
lines changed

17 files changed

+166
-48
lines changed

‎README.md‎

Lines changed: 37 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Support following features.
1010
* Deleting columns
1111
* Updating columns by specified expression(fixed value or dynamic value)
1212
* Ordering columns by specified order
13+
* Support adding column at any position
1314

1415
## Related libraries document
1516

@@ -43,10 +44,10 @@ Search files that matches conditions specified by `--dir` and `--files`.
4344
[INFO] Scanning for projects...
4445
[INFO]
4546
[INFO] -------------------< com.example:csv-bulk-commands >--------------------
46-
[INFO] Building csv-bulk-commands 0.0.2-SNAPSHOT
47+
[INFO] Building csv-bulk-commands 0.0.5-SNAPSHOT
4748
[INFO] --------------------------------[ jar ]---------------------------------
4849
[INFO]
49-
[INFO] >>> spring-boot-maven-plugin:2.5.3:run (default-cli) > test-compile @ csv-bulk-commands >>>
50+
[INFO] >>> spring-boot-maven-plugin:2.6.4:run (default-cli) > test-compile @ csv-bulk-commands >>>
5051
[INFO]
5152
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ csv-bulk-commands ---
5253
[INFO] Using 'UTF-8' encoding to copy filtered resources.
@@ -55,25 +56,22 @@ Search files that matches conditions specified by `--dir` and `--files`.
5556
[INFO] Copying 0 resource
5657
[INFO]
5758
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ csv-bulk-commands ---
58-
[INFO] Changes detected - recompiling the module!
59-
[INFO] Compiling 6 source files to /Users/xxx/git-pub/csv-bulk-commands/target/classes
59+
[INFO] Nothing to compile - all classes are up to date
6060
[INFO]
6161
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ csv-bulk-commands ---
6262
[INFO] Using 'UTF-8' encoding to copy filtered resources.
6363
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
64-
[INFO] Copying 8 resources
64+
[INFO] Copying 14 resources
6565
[INFO]
6666
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ csv-bulk-commands ---
67-
[INFO] Changes detected - recompiling the module!
68-
[INFO] Compiling 2 source files to /Users/xxx/git-pub/csv-bulk-commands/target/test-classes
67+
[INFO] Nothing to compile - all classes are up to date
6968
[INFO]
70-
[INFO] <<< spring-boot-maven-plugin:2.5.3:run (default-cli) < test-compile @ csv-bulk-commands <<<
69+
[INFO] <<< spring-boot-maven-plugin:2.6.4:run (default-cli) < test-compile @ csv-bulk-commands <<<
7170
[INFO]
7271
[INFO]
73-
[INFO] --- spring-boot-maven-plugin:2.5.3:run (default-cli) @ csv-bulk-commands ---
72+
[INFO] --- spring-boot-maven-plugin:2.6.4:run (default-cli) @ csv-bulk-commands ---
7473
[INFO] Attaching agents: []
7574
76-
7775
[Arguments]
7876
--command
7977
adding-columns, deleting-columns, updating-columns, ordering-columns
@@ -101,6 +99,10 @@ Search files that matches conditions specified by `--dir` and `--files`.
10199
delimiter character (default: ",")
102100
--ignore-escaped-enclosure
103101
whether ignore escape an enclosing character on writing (default: false)
102+
--first
103+
indicate that adding column at first position
104+
--after
105+
indicate that adding column at after position
104106
--h (--help)
105107
print help
106108
@@ -117,6 +119,28 @@ Search files that matches conditions specified by `--dir` and `--files`.
117119
001,test,1,NULL
118120
------------------------
119121
122+
e.g.) --command=adding-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item10,item11 --column-values=1,'NULL' --first
123+
------------------------
124+
item1,item2
125+
001,test
126+
------------------------
127+
↓
128+
------------------------
129+
item10,item11,item1,item2
130+
1,NULL,001,test
131+
------------------------
132+
133+
e.g.) --command=adding-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item10,item11 --column-values=1,'NULL' --after=item1
134+
------------------------
135+
item1,item2
136+
001,test
137+
------------------------
138+
↓
139+
------------------------
140+
item1,item10,item11,item2
141+
001,1,NULL,test
142+
------------------------
143+
120144
[Usage: deleting-columns]
121145
Deleting specified existing column using column-names.
122146
e.g.) --command=deleting-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item2,item9
@@ -155,11 +179,12 @@ Search files that matches conditions specified by `--dir` and `--files`.
155179
item9,item8,item2,item1
156180
foo,1,test,001
157181
------------------------
182+
158183
[INFO] ------------------------------------------------------------------------
159184
[INFO] BUILD SUCCESS
160185
[INFO] ------------------------------------------------------------------------
161-
[INFO] Total time: 4.956 s
162-
[INFO] Finished at: 2021-08-09T10:45:05+09:00
186+
[INFO] Total time: 2.376 s
187+
[INFO] Finished at: 2022-02-27T12:44:55+09:00
163188
[INFO] ------------------------------------------------------------------------
164189
```
165190

‎src/main/java/com/example/tools/AddingColumnProcessor.java‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import java.util.List;
1414
import java.util.Map;
1515
import java.util.Objects;
16+
import java.util.stream.Stream;
1617

1718
public class AddingColumnProcessor extends AbstractColumnProcessor {
1819

@@ -22,24 +23,43 @@ private AddingColumnProcessor() {
2223
// NOP
2324
}
2425

25-
void execute(List<String> columnNames, List<String> columnValues, Path file, Charset encoding, Map<String, Object> valueMappings, String delimiter, Boolean ignoreEscapedEnclosure) {
26+
void execute(String method, String target, List<String> columnNames, List<String> columnValues, Path file, Charset encoding, Map<String, Object> valueMappings, String delimiter, Boolean ignoreEscapedEnclosure) {
2627
try {
2728
final List<String> originalHeaderColumns = readHeaderColumns(file, encoding, delimiter);
2829
if (originalHeaderColumns == null) {
2930
logger.warn("Skip adding because file is empty. file:{}", file);
3031
return;
3132
}
33+
if ("after".equalsIgnoreCase(method) && (originalHeaderColumns.isEmpty() || Stream.of(target, target.toLowerCase(),
34+
target.toUpperCase()).map(x -> originalHeaderColumns.indexOf(target))
35+
.noneMatch(x -> x != -1))) {
36+
throw new IllegalArgumentException(
37+
"Cannot use the 'after' option because no header name in csv file header.");
38+
}
3239
final List<String> headerColumns = new ArrayList<>(originalHeaderColumns);
3340
Map<String, Integer> headerIndexMap = new LinkedHashMap<>();
3441
for (String column : headerColumns) {
3542
headerIndexMap.put(column, headerIndexMap.size());
3643
}
44+
45+
int addingStartIndex;
46+
if ("first".equalsIgnoreCase(method)) {
47+
addingStartIndex = 0;
48+
} else if ("after".equalsIgnoreCase(method)) {
49+
addingStartIndex =
50+
Stream.of(target, target.toLowerCase(), target.toUpperCase()).map(x -> headerColumns.indexOf(target))
51+
.filter(x -> x != -1).findAny().orElse(0) + 1;
52+
} else {
53+
addingStartIndex = headerColumns.size();
54+
}
55+
3756
Map<Integer, Boolean> containsColumnMap = new LinkedHashMap<>();
57+
int baseHeaderColumnSize = headerColumns.size();
3858
for (String columnName : columnNames) {
3959
boolean contains = headerColumns.contains(columnName);
4060
containsColumnMap.put(containsColumnMap.size(), contains);
4161
if (!contains) {
42-
headerColumns.add(columnName);
62+
headerColumns.add(addingStartIndex + (headerColumns.size() - baseHeaderColumnSize), columnName);
4363
}
4464
}
4565
List<String> validColumnValues = new ArrayList<>();
@@ -53,12 +73,16 @@ void execute(List<String> columnNames, List<String> columnValues, Path file, Cha
5373
saveLines.add(headerColumns.toArray(new String[0]));
5474
for (String[] items : lines) {
5575
List<String> valueColumns = new ArrayList<>(Arrays.asList(items));
76+
if ("last".equalsIgnoreCase(method)) {
77+
addingStartIndex = valueColumns.size();
78+
}
79+
int baseValueColumnSize = valueColumns.size();
5680
StandardEvaluationContext context = new StandardEvaluationContext();
5781
context.setVariable("_valueMappings", valueMappings);
5882
headerIndexMap.forEach((name, index) -> context.setVariable(name, valueColumns.get(index)));
5983
for (String value : validColumnValues) {
6084
Expression expression = expressionParser.parseExpression(value);
61-
valueColumns.add(Objects.toString(expression.getValue(context)));
85+
valueColumns.add(addingStartIndex + (valueColumns.size() - baseValueColumnSize), Objects.toString(expression.getValue(context)));
6286
}
6387
saveLines.add(valueColumns.toArray(new String[0]));
6488
}

‎src/main/java/com/example/tools/CsvBulkCommandsApplicationRunner.java‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,10 @@ public void run(ApplicationArguments args) throws IOException {
5858
System.out.println(" delimiter character (default: \",\")");
5959
System.out.println(" --ignore-escaped-enclosure");
6060
System.out.println(" whether ignore escape an enclosing character on writing (default: false)");
61+
System.out.println(" --first");
62+
System.out.println(" indicate that adding column at first position");
63+
System.out.println(" --after");
64+
System.out.println(" indicate that adding column at after position");
6165
System.out.println(" --h (--help)");
6266
System.out.println(" print help");
6367
System.out.println();
@@ -74,6 +78,28 @@ public void run(ApplicationArguments args) throws IOException {
7478
System.out.println(" 001,test,1,NULL");
7579
System.out.println(" ------------------------");
7680
System.out.println();
81+
System.out.println(" e.g.) --command=adding-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item10,item11 --column-values=1,'NULL' --first");
82+
System.out.println(" ------------------------");
83+
System.out.println(" item1,item2");
84+
System.out.println(" 001,test");
85+
System.out.println(" ------------------------");
86+
System.out.println(" ↓");
87+
System.out.println(" ------------------------");
88+
System.out.println(" item10,item11,item1,item2");
89+
System.out.println(" 1,NULL,001,test");
90+
System.out.println(" ------------------------");
91+
System.out.println();
92+
System.out.println(" e.g.) --command=adding-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item10,item11 --column-values=1,'NULL' --after=item1");
93+
System.out.println(" ------------------------");
94+
System.out.println(" item1,item2");
95+
System.out.println(" 001,test");
96+
System.out.println(" ------------------------");
97+
System.out.println(" ↓");
98+
System.out.println(" ------------------------");
99+
System.out.println(" item1,item10,item11,item2");
100+
System.out.println(" 001,1,NULL,test");
101+
System.out.println(" ------------------------");
102+
System.out.println();
77103
System.out.println("[Usage: deleting-columns]");
78104
System.out.println(" Deleting specified existing column using column-names.");
79105
System.out.println(" e.g.) --command=deleting-columns --dir=src/test/resources/data --files=xxx.csv,yyy.csv --column-names=item2,item9");
@@ -166,22 +192,38 @@ public void run(ApplicationArguments args) throws IOException {
166192
Boolean ignoreEscapedEnclosure = args.containsOption("ignore-escaped-enclosure") &&
167193
Boolean.parseBoolean(args.getOptionValues("ignore-escaped-enclosure").stream().findFirst().orElse(null));
168194

169-
LOGGER.info("Start. command:{} dir:{} files:{} column-names:{} column-values:{} encoding:{} value-mappings:{} delimiter:{} ignore-escaped-enclosure:{}",
170-
command, dir, files, columnNames, columnValues, encoding, valueMappings, delimiter, ignoreEscapedEnclosure);
195+
final String addingMethod;
196+
final String addingTarget;
197+
if (args.containsOption("first")) {
198+
addingMethod = "first";
199+
addingTarget = null;
200+
} else if (args.containsOption("after")) {
201+
addingMethod = "after";
202+
addingTarget = args.getOptionValues("after").stream().findFirst().orElse(null);
203+
if (addingTarget == null) {
204+
throw new IllegalArgumentException("'after' is required.");
205+
}
206+
} else {
207+
addingMethod = "last";
208+
addingTarget = null;
209+
}
210+
211+
LOGGER.info("Start. command:{} dir:{} files:{} column-names:{} column-values:{} encoding:{} value-mappings:{} delimiter:{} ignore-escaped-enclosure:{} adding-method:{} adding-target:{}",
212+
command, dir, files, columnNames, columnValues, encoding, valueMappings, delimiter, ignoreEscapedEnclosure, addingMethod, addingTarget);
171213

172214
Files.walk(Paths.get(dir))
173215
.filter(Files::isRegularFile)
174216
.filter(file -> files.stream().anyMatch(x -> file.toString().replace('\\', '/').endsWith(x)))
175-
.sorted().forEach(file -> execute(command, columnNames, columnValues, file, encoding, valueMappings, delimiter, ignoreEscapedEnclosure));
217+
.sorted().forEach(file -> execute(command, columnNames, columnValues, file, encoding, valueMappings, delimiter, ignoreEscapedEnclosure, addingMethod, addingTarget));
176218

177219
LOGGER.info("End.");
178220
}
179221

180-
private void execute(String command, List<String> columnNames, List<String> columnValues, Path file, Charset encoding, Map<String, Object> valueMappings, String delimiter, Boolean ignoreEscapedEnclosure) {
222+
private void execute(String command, List<String> columnNames, List<String> columnValues, Path file, Charset encoding, Map<String, Object> valueMappings, String delimiter, Boolean ignoreEscapedEnclosure, String addingMethod, String addingTarget) {
181223
LOGGER.info("processing file:{}", file);
182224
switch (command) {
183225
case "adding-columns":
184-
AddingColumnProcessor.INSTANCE.execute(columnNames, columnValues, file, encoding, valueMappings, delimiter, ignoreEscapedEnclosure);
226+
AddingColumnProcessor.INSTANCE.execute(addingMethod, addingTarget, columnNames, columnValues, file, encoding, valueMappings, delimiter, ignoreEscapedEnclosure);
185227
break;
186228
case "deleting-columns":
187229
DeletingColumnProcessor.INSTANCE.execute(columnNames, file, encoding, delimiter, ignoreEscapedEnclosure);

‎src/test/java/com/example/tools/CsvBulkCommandsApplicationRunnerTests.java‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,26 +11,38 @@ class CsvBulkCommandsApplicationRunnerTests {
1111

1212
@Test
1313
void addingColumns() throws IOException {
14-
String[] args = {"--command=adding-columns", "--files=aaa.csv,bbb.csv", "--column-names=x,y", "--column-values=100*0.1", "--dir=src/test/resources/data"};
14+
String[] args = {"--command=adding-columns", "--files=aaa.csv,bbb.csv", "--column-names=x,y", "--column-values=100*0.1,'abc'", "--dir=target/test-classes/data"};
15+
runner.run(new DefaultApplicationArguments(args));
16+
}
17+
18+
@Test
19+
void addingColumns2() throws IOException {
20+
String[] args = {"--command=adding-columns", "--files=fff.csv", "--column-names=x,y", "--column-values=100*0.1,'abc'", "--dir=target/test-classes/data", "--first"};
21+
runner.run(new DefaultApplicationArguments(args));
22+
}
23+
24+
@Test
25+
void addingColumns3() throws IOException {
26+
String[] args = {"--command=adding-columns", "--files=ggg.csv", "--column-names=x,y", "--column-values=100*0.1,'abc'", "--dir=target/test-classes/data", "--after=a"};
1527
runner.run(new DefaultApplicationArguments(args));
1628
}
1729

1830
@Test
1931
void deletingColumns() throws IOException {
20-
String[] args = {"--command=deleting-columns", "--files=aaa.csv,bbb.csv", "--column-names=c", "--dir=src/test/resources/data"};
32+
String[] args = {"--command=deleting-columns", "--files=ccc.csv", "--column-names=b", "--dir=target/test-classes/data"};
2133
runner.run(new DefaultApplicationArguments(args));
2234
}
2335

2436
@Test
2537
void updatingColumns() throws IOException {
26-
String[] args = {"--command=updating-columns", "--files=aaa.csv,bbb.csv", "--column-names=y,x", "--column-values='NULL','CURRENT_TIMESTAMP'", "--dir=src/test/resources/data"};
38+
String[] args = {"--command=updating-columns", "--files=ddd.csv", "--column-names=y,x", "--column-values='NULL','CURRENT_TIMESTAMP'", "--dir=target/test-classes/data"};
2739
runner.run(new DefaultApplicationArguments(args));
2840
}
2941

3042

3143
@Test
3244
void orderingColumns() throws IOException {
33-
String[] args = {"--command=ordering-columns", "--files=aaa.csv,bbb.csv", "--column-names=a,b,c,y,x", "--dir=src/test/resources/data"};
45+
String[] args = {"--command=ordering-columns", "--files=eee.csv", "--column-names=a,b,c,y,x", "--dir=target/test-classes/data"};
3446
runner.run(new DefaultApplicationArguments(args));
3547
}
3648

‎src/test/resources/data/XxxTest/expected/aaa.csv‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
a,b,y,x
2-
1,2,NULL,CURRENT_TIMESTAMP
3-
4,5,NULL,CURRENT_TIMESTAMP
1+
a,b
2+
1,2
3+
4,5

‎src/test/resources/data/XxxTest/expected/bbb.csv‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
1-
a,b,y,x
2-
3,2,NULL,CURRENT_TIMESTAMP
3-
6,5,NULL,CURRENT_TIMESTAMP
1+
a,b
2+
3,2
3+
6,5

‎src/test/resources/data/XxxTest/expected/ccc.csv‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
a,b,c
2+
3,2,aa
3+
6,5,bb

‎src/test/resources/data/XxxTest/expected/ddd.csv‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
a,x,c,y
2+
3,2,aa,111
3+
6,5,bb,222

‎src/test/resources/data/XxxTest/expected/eee.csv‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
a,y,c,x,b
2+
3,2,aa,111,uuu
3+
6,5,bb,222,yyy

‎src/test/resources/data/XxxTest/expected/fff.csv‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
a,b
2+
1,2
3+
4,5

0 commit comments

Comments
 (0)