(WINDOWS)解决IDEA运行MapReduce程序没有日志问题

背景

问题:MapReduce项目 可以运行成功,但是控制台只有几条信息,说明项目没有配置log4j,在开发的过程中,我们需要更详细的日志信息来定位问题和查看整个过程。

1
2
3
4
5
6
7
8
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Process finished with exit code 0

项目打包成jar包,放到hdfs上运行,是有完整的日志信息的,可以看到整个MapRuduce的执行过程。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
2021-12-16 15:39:45,000 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-12-16 15:39:46,374 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2021-12-16 15:39:47,293 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-12-16 15:39:47,317 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1639471863570_0002
Thu Dec 16 15:39:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-12-16 15:39:48,301 INFO mapreduce.JobSubmitter: number of splits:2
2021-12-16 15:39:48,601 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1639471863570_0002
2021-12-16 15:39:48,603 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-12-16 15:39:48,910 INFO conf.Configuration: resource-types.xml not found
2021-12-16 15:39:48,910 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-12-16 15:39:49,056 INFO impl.YarnClientImpl: Submitted application application_1639471863570_0002
2021-12-16 15:39:49,108 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1639471863570_0002/
2021-12-16 15:39:49,109 INFO mapreduce.Job: Running job: job_1639471863570_0002
2021-12-16 15:40:00,420 INFO mapreduce.Job: Job job_1639471863570_0002 running in uber mode : false
2021-12-16 15:40:00,424 INFO mapreduce.Job: map 0% reduce 0%
2021-12-16 15:40:15,639 INFO mapreduce.Job: map 100% reduce 0%
2021-12-16 15:40:23,706 INFO mapreduce.Job: map 100% reduce 100%
2021-12-16 15:40:23,717 INFO mapreduce.Job: Job job_1639471863570_0002 completed successfully
2021-12-16 15:40:23,832 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=106
FILE: Number of bytes written=705400
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=156
HDFS: Number of bytes written=80
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=24451
Total time spent by all reduces in occupied slots (ms)=5384
Total time spent by all map tasks (ms)=24451
Total time spent by all reduce tasks (ms)=5384
Total vcore-milliseconds taken by all map tasks=24451
Total vcore-milliseconds taken by all reduce tasks=5384
Total megabyte-milliseconds taken by all map tasks=25037824
Total megabyte-milliseconds taken by all reduce tasks=5513216
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=92
Map output materialized bytes=112
Input split bytes=156
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=112
Reduce input records=4
Reduce output records=4
Spilled Records=8
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=515
CPU time spent (ms)=3130
Physical memory (bytes) snapshot=507174912
Virtual memory (bytes) snapshot=8157011968
Total committed heap usage (bytes)=307437568
Peak Map Physical memory (bytes)=199159808
Peak Map Virtual memory (bytes)=2718306304
Peak Reduce Physical memory (bytes)=109441024
Peak Reduce Virtual memory (bytes)=2720411648
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=80

在此,我的项目有log4j依赖,但缺少log4j配置文件。

1
2
3
4
5
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
<version>2.10.0</version>
</dependency>

添加log4j配置文件

通过网上搜查得知,hadoop目录下hadoop/etc/hadoop是有自带的log4j.properties配置文件的。把log4j.properties 复制到project->src->main->resources下面,再执行程序。显示如下:

1
2
3
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

大概意思就是需要SLF4J依赖,于是搜索如何添加SLF4J依赖:

slf4j打印日志必须的三个依赖包

slf4j假设使用log4j做为底层日志工具,运行以上程序需要三个包:

  • log4j-1.2.xx.jar、

  • slf4j-api-x.x.x.jar、

  • slf4j-log4j12-x.x.x.jar

经调试,直接把原来org.apache.logging.log4j依赖,替换成上面3个包的依赖:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.21</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.21</version>
</dependency>

然后运行程序,控制台输出日志与服务器上运行输出的信息的大致一样。